Q1. (Ridge Regression, Theoretical Understanding, 10 marks)
Consider the linear regression model $y = X\beta + \epsilon$ for $X \in \mathbb{R}^{n \times p}$ by $l_2$-regularized least squares:
$$\min_{\beta} (y - X\beta)^T(y - X\beta) + \lambda ||\beta||_2^2$$
where $\lambda > 0$ is the regularization parameter.
a) (2 marks) Derive the closed-form solution $\hat{\beta}^{Ridge}(\lambda)$ in an analytical way.
b) (2 marks) When $\epsilon \sim N(0, \sigma^2I_n)$, prove that the bias and covariance of $\hat{\beta}^{Ridge}(\lambda)$ are
$$Bias(\hat{\beta}^{Ridge}(\lambda)) = -\lambda(X^TX + \lambda I)^{-1}\beta$$
$$Cov(\hat{\beta}^{Ridge}(\lambda)) = \sigma^2(X^TX + \lambda I)^{-1}X^TX(X^TX + \lambda I)^{-1}$$
b) (2 marks) Define the MSE (mean squared error) of the ridge estimator by
$$MSE(\hat{\beta}^{Ridge}(\lambda)) = E\left[(\hat{\beta}^{Ridge}(\lambda) - \beta)^T(\hat{\beta}^{Ridge}(\lambda) - \beta)\right].$$
Re-express it in terms of $Bias(\hat{\beta}^{Ridge}(\lambda))$ and $Cov(\hat{\beta}^{Ridge}(\lambda))$.
(1)