(Covariance estimation; 3+4+3 pts) We study estimation of the covariance structure of data in this problem. Consider i.i.d. Gaussian vectors X1,...,Xn ~ N(0,E) in Rd with a covariance matrix Σ ∈ R^d×d, where n ≤ d. The empirical covariance matrix is defined as
Σ_hat = (1/n) ∑_(i=1)^n x_i x_i^T
which is an unbiased estimator of Σ. We aim to bound the estimation error in the operator norm ||Σ_hat - Σ||.
(a) Suppose that Σ = I_d. Fix x,y ∈ R^d where R^d is the 1/4-net given in Problem 2(a). Using the "polarization" identity
(x^T x)(x^T y)=[(x+y)^T(x+y)]^2-[(x-y)^T(x-y)]^2
show that
||Σ_hat - Σ||_2 ≤ (1/4)(||x||_2^2 - ||y||_2^2)
for some properly defined random variables Z, W ~ N(0,Σ).
(b) Suppose that Σ = I_d. Use the tail bound (no need to prove it)
P{|Z-E[Z]|^2 ≥ nt+2t} ≤ e^(-t)
to derive a tail bound for x^T(Σ_hat - Σ)y. Then combine this with Problem 2(a) to show that with probability at least 0.99, ||Σ_hat - Σ||_2 ≤ H_2(Σ) + ||I_d||_S^2V/n.
(Remark: The assumed tail bound for a χ^2 random variable can be proved using the Chernoff bound similar to what we did for sub-Gaussian random variables.)