3. (MATH2831 and MATH2931) In this question we consider derivation of the C_p statistic for model selection discussed in lectures. Consider fitting the proposed general linear model and write ?_i for the fitted value at x_i and MSE(?_i) for its mean squared error. Recall that if the error variance ?^2 is known, then an estimate of J_p = sum_i MSE(?_i) / ?^2 = sum_i Var(?_i) / ?^2 + sum_i Bias^2(?_i) / ?^2 is p + (n - p)(hat?^2 - ?^2) / ?^2 (1) where hat?^2 is the estimate of the residual error variance for the proposed model and p is the number of parameters. Substituting hat?_F^2 (the estimated error variance based on a model including all predictors) for ?^2 in this expression gives Mallow's C_p. You now have to provide a justification for (1). In what follows, we let X_1 be the design matrix for the proposed model, and X_2 be the design matrix for the true model. Let ?^(1) denote the vector of parameters in the proposed model and ?^(2) be the vector of parameters for the true model. Write b^(1) for the least squares estimator of ?^(1) and b^(2) for the least squares estimator of ?^(2). (a) Writing ? for the vector of fitted values for the proposed model and observing that ? = X_1 (X_1^T X_1)^{-1} X_1^T y = H_1 y, show that sum_i Var(?_i) = ?^2 tr(H_1), where tr(A) denotes the trace of A and H_1 = X_1 (X_1^T X_1)^{-1} X_1^T denotes the hat matrix corresponding to the proposed model. By using the rules given in lectures about matrix traces, deduce that sum_i Var(?_i) / ?^2 = p.
Added by Jill B.
Close
Step 1
Since y = Xβ(1), we can substitute this into the variance expression to get Var(y) = Var(Xb(1)). We also know that the least squares estimator b(1) is given by b(1) = (X'X)^(-1)X'y. Substituting this into the variance expression, we get Var(y) = Show more…
Show all steps
Your feedback will help us improve your experience
Sri K and 96 other Intro Stats / AP Statistics educators are ready to help you.
Ask a new question
Labs
Want to see this concept in action?
Explore this concept interactively to see how it behaves as you change inputs.
Key Concepts
Recommended Videos
(Weighted Least Squares) Suppose that in the model $y_{i}=\beta_{0}+\beta_{1} x_{i}+e_{i},$ the errors have mean zero and are independent, but $\operatorname{Var}\left(e_{i}\right)=\rho_{i}^{2} \sigma^{2},$ where the $\rho_{i}$ are known constants, so the errors do not have equal variance. This situation arises when the $y_{i}$ are averages of several observations at $x_{i}$; in this case, if $y_{i}$ is an average of $n_{i}$ independent observations, $\rho_{i}^{2}=1 / n_{i}$ (why?). Because the variances are not equal, the theory developed in this chapter does not apply; intuitively, it seems that the observations with large variability should influence the estimates of $\beta_{0}$ and $\beta_{1}$ less than the observations with small variability. The problem may be transformed as follows: $$ \rho_{i}^{-1} y_{i}=\rho_{i}^{-1} \beta_{0}+\rho_{i}^{-1} \beta_{1} x_{i}+\rho_{i}^{-1} e_{i} $$ or $$ z_{i}=u_{i} \beta_{0}+v_{i} \beta_{1}+\delta_{i} $$ where $$ u_{i}=\rho_{i}^{-1} \quad v_{i}=\rho_{i}^{-1} x_{i} \quad \delta_{i}=\rho_{i}^{-1} e_{i} $$ a. Show that the new model satisfies the assumptions of the standard statistical model. b. Find the least squares estimates of $\beta_{0}$ and $\beta_{1}$ c. Show that performing a least squares analysis on the new model, as was done in part (b), is equivalent to minimizing $$ \sum_{i=1}^{n}\left(y_{i}-\beta_{0}-\beta_{1} x_{i}\right)^{2} \rho_{i}^{-2} $$ This is a weighted least squares criterion; the observations with large variances are weighted less. d. Find the variances of the estimates of part (b).
Introduction Suppose you want to fit a linear regression model Y = X̠ + ̵ where Y is a nx1 vector of response variables, X is an nxp design matrix, ̠ is a px1 coefficient vector and ̵ is an nx1 vector of independent N(0, ̓²) random variables. Suppose in addition that you cannot observe all of the elements of X directly: instead, you observe V=X+U where the elements of U are themselves independent normal random variables with zero mean. The variance of the (i,j)th element of U is ̢Ⲝ²: note that this does not depend on i. One way to interpret this is: the entries in the jth column of the design matrix are all subject to independent measurement errors, which are normally distributed with variance ̢Ⲝ². In this situation, to fit the regression model it is natural to use the usual least-squares procedure with the observed design matrix V in place of the true values X. It can be shown, however, that this produces biased estimates of the coefficient vector ̠. A corrected estimator can be defined, however, as ̠̃ = (V'V – D)⁻¹V'Y, where D is a pxp diagonal matrix with jth diagonal element cⲜⲜ = n̢Ⲝ². The covariance matrix of this estimate is ̓²(V'V – D)⁻¹V'V(V'V – D)⁻¹ . Standard errors of the estimated parameters can then be calculated as the square roots of the diagonal elements of this covariance matrix. For this question, we will assume that the value of ̓² is known. Your task Without using the R command lm() or anything similar, write an R function called lm.adjusted(), to implement the corrected estimator as described above. The arguments to your function should be y, a vector of responses corresponding to Y in the description above; V, a design matrix corresponding to V in the description (note that V is in UPPER CASE, whereas y is in lower case: this is not clear on some browsers); sigsq, the known value of ̓²; and tausq, a vector containing the values of {̢Ⲝ²}. The default value for tausq should be a vector of zeroes. You may assume that the matrix V'V-D is invertible. Your function should return a list containing components betahat (the estimated coefficient vector) and sebeta (a vector of estimated standard errors for the coefficients).
Sri K.
Use the data set 401 $\mathrm{KSUBS}$ for this exercise. (i) Using OLS, estimate a linear probability model for $e 401 k,$ using as explanatory variables inc, inc $^{2},$ age, $a g e^{2},$ and male. Obtain both the usual OLS standard errors and the heteroskedasticity- robust versions. Are there any important differences? (ii) In the special case of the White test for heteroskedasticity, where we regress the squared OLS residuals on a quadratic in the OLS fitted values, $\hat{u}_{i}^{2}$ on $\hat{y}_{i}, \hat{y}_{i}^{2}, i=1, \ldots, n,$ argue that the probability limit of the coefficient on $\hat{y}_{i}$ should be one, the probability limit of the coefficient on $\hat{y}_{i}^{2}$ should be $-1,$ and the probability limit of the intercept should be zero. $\{$ Hint: Remember that $\operatorname{Var}(y | x_{1}, \ldots, x_{k})=p(\mathbf{x})[1-p(\mathbf{x})],$ where $p(\mathbf{x})=\beta_{0}+\beta_{1} x_{1}+\ldots+\beta_{k} x_{k} \}$ (iii) For the model estimated from part (i), obtain the White test and see if the coefficient estimates roughly correspond to the theoretical values described in part (ii). (iv) After verifying that the fitted values from part (i) are all between zero and one, obtain the weighted least squares estimates of the linear probability model. Do they differ in important ways from the OLS estimates?
Recommended Textbooks
Elementary Statistics a Step by Step Approach
The Practice of Statistics for AP
Introductory Statistics
Transcript
Watch the video solution with this free unlock.
EMAIL
PASSWORD