Optimal linear shrinkage

ProofAdvanced⏱ 1 hourHigh-Dim Statisticsshrinkageledoit-wolfbias-variance

Problem

Consider $\hat\Sigma_\alpha = (1-\alpha)\hat\Sigma+\alpha\hat\mu I_p$ with $\hat\mu = \operatorname{tr}(\hat\Sigma)/p$ . Assume $\Sigma = I_p$ .

Write $\E\|\hat\Sigma_\alpha - I\|_F^2$ as a quadratic in $\alpha$ .
Find the minimising $\alpha^\star$ and show it lies in $(0,1)$ whenever $p\ge2$ .
Explain why the unbiased estimator ( $\alpha=0$ ) is inadmissible in this loss.

Write a complete argument. Every inequality you use should be one you could state precisely.

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
The bias–variance decomposition is an identity	Learning Theory	Proof	Beginner	15 min
Measure the covariance estimation rate run	High-Dim Statistics	Coding	Intermediate	1 hour
Ridge regression in the SVD basis	Classical ML	Proof	Intermediate	30 min
Sample complexity of covariance estimation	High-Dim Statistics	Numerical	Intermediate	30 min
There is always a ridge parameter beating least squares abyss	Classical ML	Proof	Advanced	1 hour