Suppose that we have a model y_(i)=eta x_(i)+epsilon_(i) (i=1,dots,n) where ar{y}=(1/n)sum_(i=1)^n y_(i)=0, ar{x}=(1/n)sum_(i=1)^n x_(i)=0, and epsilon_(i) is distributed normally with mean 0 and variance sigma^2; that is, epsilon_(i)∼N(0,sigma^2). Furthermore, epsilon_(1),epsilon_(2),dots,epsilon_(n) are independently distributed, and the x_(i)'s (i=1,dots,n) are non-random.
(a) The OLS estimator for eta minimizes the Sum of Squared Residuals:
hat{eta}=argmin_{eta}[sum_(i=1)^n (y_(i)-eta x_(i))^2]
Take the first-order condition to show that
hat{eta}=(sum_(i=1)^n x_(i)y_(i))/(sum_(i=1)^n x_(i)^2).
(b) Assume E[epsilon_(i)|eta]=0 for all i=1,dots,n. Show that
hat{eta}=eta+(sum_(i=1)^n x_(i)epsilon_(i))/(sum_(i=1)^n x_(i)^2)
What is E[hat{eta}|eta] and Var(hat{eta}|eta)? Use this to show that, conditional on eta, hat{eta} has the following distribution:
hat{eta}|eta∼N(eta,(sigma^2)/(sum_(i=1)^n x_(i)^2)).
(c) Suppose we believe that eta is distributed normally with mean 0 and variance (sigma^2)/(lambda); that is, eta∼N(0,(sigma^2)/(lambda)). Additionally, assume that eta is independent of epsilon_(i) for all i=1,dots,n. Compute the mean and variance of hat{eta}. That is, what is E[hat{eta}] and Var(hat{eta])?
(Hint: you might find useful: E[w_(1)]=E[E[w_(1)|w_(2)]] and Var(w_(1))=E[Var(w_(1)|w_(2))]+Var(E[w_(1)|w_(2)]) for any random variables w_(1) and w_(2).)
(d) Since everything is normally distributed, it turns out that
E[eta|hat{eta}]=E[eta]+(Cov(eta,hat{eta}))/Var(hat{eta})*(hat{eta}-E[hat{eta}]).
Let hat{eta}^(RR)=E[eta|hat{eta}]. Compute Cov(eta,hat{eta}) and use the value of E[eta] along with the values of E[hat{eta}], Cov(eta,hat{eta}), and Var(hat{eta}) you have computed to show that
hat{eta}^(RR)=E[eta|hat{eta)]=(sum_(i=1)^n x_(i)^2)/(sum_(i=1)^n x_(i)^2+lambda)*hat{eta}
(Hint: Cov(w_(1),w_(2))=E[(w_(1)-E[w_(1)])(w_(2)-E[w_(2)])] and E[w_(1)w_(2)]=E[w_(1)E[w_(2)|w_(1)]] for any random variables w_(1) and w_(2))
(e) Does hat{eta}^(RR) increase or decrease as lambda increases? How does this relate to eta being distributed N(0,(sigma^2)/(lambda))?
