The bias–variance decomposition is an identity

ProofBeginner⏱ 15 minLearning Theorybias-varianceidentitysquared loss

Theory: Bias–Variance, Interpolation and Double Descent →

Problem

Prove that for $y = f^\star(x)+\varepsilon$ with $\E[\varepsilon\mid x]=0$ , $\Var(\varepsilon\mid x)=\sigma^2$ , and an estimator $\hat f_S$ trained on a random sample,

\E\big[(y-\hat f_S(x))^2\big] = \sigma^2 + \big(\E_S\hat f_S(x)-f^\star(x)\big)^2 + \Var_S\big(\hat f_S(x)\big).

State explicitly which hypotheses are used and which are not.

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Gradient descent finds the minimum-norm interpolator	Learning Theory	Proof	Intermediate	30 min
Optimal linear shrinkage	High-Dim Statistics	Proof	Advanced	1 hour
Reproduce double descent run	Learning Theory	Coding	Advanced	3 hours
The interpolation peak is a conditioning phenomenon	Learning Theory	Proof	Advanced	30 min
There is always a ridge parameter beating least squares abyss	Classical ML	Proof	Advanced	1 hour