Ridge regression in the SVD basis

ProofIntermediate⏱ 30 minClassical MLridgesvdshrinkageeffective dof

Theory: Linear, Ridge and Lasso Regression →

Problem

Express $\hat\beta_\lambda = (X^\top X+\lambda I)^{-1}X^\top y$ in terms of the SVD $X = U\Sigma V^\top$ , and show that the fitted values shrink the $j$ -th singular direction by $\sigma_j^2/(\sigma_j^2+\lambda)$ . Compute the effective degrees of freedom and its limits.

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Least squares as orthogonal projection	Classical ML	Proof	Foundation	15 min
Implement kernel ridge regression run	Classical ML	Coding	Intermediate	1 hour
Implement ridge and find the optimal lambda run	Classical ML	Coding	Intermediate	1 hour
Prove the Gauss–Markov theorem	Classical ML	Proof	Intermediate	30 min
Optimal linear shrinkage	High-Dim Statistics	Proof	Advanced	1 hour