Skip to content
nonconvexcard

Deep Double Descent: Where Bigger Models and More Data Hurt

Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, Ilya Sutskever · 2020 · ICLR

Learning TheoryDL Theorydouble descentinterpolationoverparameterisationeffective model complexity
Read the paper

The problem

The classical U-shaped risk curve predicts that risk rises monotonically once a model can interpolate, which contradicts the observed behaviour of large neural networks.

The contribution

Systematic demonstration of a second descent in test error past the interpolation threshold, across model size, dataset size and training time, unified by an effective-model-complexity variable.

What problem does this paper solve?

It establishes, at scale and across three axes, that test error is not U-shaped in capacity for deep networks — resolving a persistent mismatch between classical theory and practice into a reproducible phenomenon with a described shape.

What mathematical objects are introduced?

Effective model complexity (EMC): the largest sample size on which the training procedure achieves approximately zero training error. The paper's organising claim is that the peak in test error occurs when EMC ≈n\approx n, whichever parameter is varied to get there.

What assumptions are made?

That the interpolation threshold is the relevant control variable, and that the same mechanism explains model-wise, sample-wise and epoch-wise double descent. The last is the boldest claim and the least directly supported.

What equations matter?

None derived. The paper is experimental. The relevant analysis is the linear model treated on the double-descent page, where the variance term is σ2tr⁡((X⊤X)−1Σ)\sigma^2\tr((X^\top X)^{-1}\Sigma) and diverges as λmin⁡(X⊤X)→0\lambda_{\min}(X^\top X)\to0 at p=np=n.

What theorem or result is important?

No theorem. The empirical results: (i) test error peaks at the interpolation threshold and then falls; (ii) more data can hurt at fixed model size, when the extra data moves the model towards the peak; (iii) the peak grows with label noise and shrinks with explicit regularisation.

What proof techniques are used?

None; controlled experiments with careful isolation of the varied quantity.

What prior mathematics is required?

The bias–variance identity, the conditioning of random design matrices (Marchenko–Pastur), and the minimum-norm interpolator.

Limitations

The mechanism is not established for deep networks — EMC is a descriptive variable, not an explanation. The results are sensitive to label noise, and adding optimal regularisation removes the peak, which suggests the phenomenon is partly an artefact of insufficiently regularised training rather than a fundamental feature. And the paper describes a risk curve; it does not explain why the interpolating solution generalises.

Why it mattered

It made "bigger is sometimes worse, and then better again" a fact to be explained rather than an anomaly to be dismissed, and it provided the experimental grounding that the linear-model theory of Hastie et al. and Bartlett et al. then matched.

What came next