Scaling Laws for Neural Language Models

What problem does this paper solve?

Predictability. Before this work, whether a $10\times$ larger model would be better was an empirical question requiring the experiment. The paper turns it into an extrapolation.

What mathematical objects are introduced?

Power-law fits

in non-embedding parameters, tokens and compute, together with a joint form interpolating between the parameter- and data-limited regimes.

What assumptions are made?

That the fitted functional form extrapolates; that the learning-rate schedule and other hyperparameters are handled consistently across scales. The second assumption is where the analysis was later found wanting.

What equations matter?

The joint fit and the compute-optimal allocation derived from it. The paper's conclusion was $N^\star\propto C^{0.73}$, $D^\star\propto C^{0.27}$ — strongly favouring larger models over more data. The Lagrange computation behind such a conclusion is on the scaling page.

What theorem or result is important?

No theorem. The empirical findings: smooth power laws with no visible breaks; weak dependence on depth, width and aspect ratio at fixed parameter count; larger models are more sample-efficient.

What proof techniques are used?

Curve fitting across a systematically varied grid of runs.

What prior mathematics is required?

Log–log regression; constrained optimisation for the allocation rule; the entropy floor to interpret the irreducible term.

Limitations

The compute-optimal conclusion was superseded: Hoffmann et al. showed the fit was biased by the use of a fixed learning-rate schedule that penalised runs trained on more tokens, and refitting gives $N^\star,D^\star\propto C^{1/2}$ — an order-of-magnitude different recommendation. The laws also describe loss, not capability, and the exponents are tokeniser- and dataset-dependent.

Why it mattered

It made large-scale training planable and is the intellectual basis for the decision to build very large models. That its headline allocation recommendation was later corrected is part of the lesson: the functional form has held up, the fitted constants were methodology-dependent, and the whole enterprise remains an empirical extrapolation with no derivation.