There is always a ridge parameter beating least squares

ProofAdvanced⏱ 1 hourClassical MLridgeadmissibilitybias-variancejames-stein

Theory: Linear, Ridge and Lasso Regression →

Problem

Prove that $m(\lambda) = \E\|\hat\beta_\lambda-\beta^\star\|_2^2$ satisfies $m'(0) = -2\sigma^2\operatorname{tr}\big((X^\top X)^{-2}\big) < 0$ , so least squares is inadmissible. Identify the orders in $\lambda$ of the bias and variance contributions and explain why that asymmetry is the whole content of the result.

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Least squares as orthogonal projection	Classical ML	Proof	Foundation	15 min
The bias–variance decomposition is an identity	Learning Theory	Proof	Beginner	15 min
Implement ridge and find the optimal lambda run	Classical ML	Coding	Intermediate	1 hour
Prove the Gauss–Markov theorem	Classical ML	Proof	Intermediate	30 min
Ridge regression in the SVD basis	Classical ML	Proof	Intermediate	30 min