Why ordinary least squares degrades as p approaches n
ProofAdvanced⏱ 30 minRandom Matricesleast squaresconditioningvariancemarchenko-pastur
Problem
Let with and having i.i.d. entries, . Show that the expected squared estimation error of ordinary least squares is
and use the Marchenko–Pastur law to argue that this behaves like for large . Evaluate at and .
Write a complete argument. Every inequality you use should be one you could state precisely.
Solution
Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.
Related problems
All problems| Solved | Problem | Field | Type | Difficulty | Time |
|---|---|---|---|---|---|
| Least squares as orthogonal projection | Classical ML | Proof | 15 min | ||
| Marchenko–Pastur edges and conditioning | Random Matrices | Numerical | 15 min | ||
| The Δ²/12 noise variance | Quantization | Proof | 15 min | ||
| Frobenius versus operator norm for random matrices | Random Matrices | Numerical | 30 min | ||
| Implement ridge and find the optimal lambda run | Classical ML | Coding | 1 hour |