Derive the compute-optimal allocation

ProofIntermediate⏱ 30 minLLMsscaling lawschinchillalagrangeallocation

Theory: Scaling Laws, Fine-Tuning and In-Context Learning →

Problem

Given $\mathcal L(N,D) = E + AN^{-\alpha}+BD^{-\beta}$ and $C = 6ND$ , derive $N^\star(C)$ and $D^\star(C)$ , and evaluate the tokens-per-parameter ratio for $\alpha=\beta$ .

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
LoRA parameter accounting	LLMs	Numerical	Beginner	15 min
The 6ND FLOP estimate	LLMs	Numerical	Beginner	15 min
Mixture-of-experts sparsity accounting	LLMs	Numerical	Intermediate	30 min
The irreducible loss is the conditional entropy	LLMs	Proof	Intermediate	30 min
Derive the optimal weight update abyss	Quantization	Proof	Advanced	1 hour