Least squares as orthogonal projection

ProofFoundation⏱ 15 minClassical MLleast squaresnormal equationsprojection

Theory: Linear, Ridge and Lasso Regression →

Problem

Derive the normal equations from the first-order condition, and show that the hat matrix $H = X(X^\top X)^{-1}X^\top$ is the orthogonal projector onto $\operatorname{col}(X)$ . Verify that $\operatorname{tr}(H) = p$ and interpret.

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Characterising orthogonal projectors	Foundations	Proof	Intermediate	30 min
Conditional expectation minimises mean squared error	Probability	Proof	Intermediate	30 min
Implement ridge and find the optimal lambda run	Classical ML	Coding	Intermediate	1 hour
Prove the Gauss–Markov theorem	Classical ML	Proof	Intermediate	30 min
Ridge regression in the SVD basis	Classical ML	Proof	Intermediate	30 min