Open: why are neural scaling laws power laws?

ProofResearch⏱ ResearchLLMsLearning Theoryopen problemscaling lawsexponentspower law

Theory: Scaling Laws, Fine-Tuning and In-Context Learning →

Problem

Empirically, test loss of a language model follows $L(N) \approx L_\infty + (N_c/N)^{\alpha_N}$ in parameters and an analogous law in data, over many orders of magnitude, with exponents that are stable within a family and vary across modalities.

Derive the functional form from a stated model of the data and the architecture, and predict $\alpha_N$ from measurable properties of the data distribution.

This is an open problem. Nobody has a solution, and none is hidden below — what is recorded is the state of the argument. Progress on it would be publishable.

What is known

This problem is open. What follows is an account of what has been proved, what has only been measured, and where the argument stops — not a solution, because there is not one.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
LoRA parameter accounting	LLMs	Numerical	Beginner	15 min
The 6ND FLOP estimate	LLMs	Numerical	Beginner	15 min
Derive the compute-optimal allocation	LLMs	Proof	Intermediate	30 min
Mixture-of-experts sparsity accounting	LLMs	Numerical	Intermediate	30 min
The irreducible loss is the conditional entropy	LLMs	Proof	Intermediate	30 min