The 6ND FLOP estimate

NumericalBeginner⏱ 15 minLLMsflopscompute accountingtraining cost

Theory: Scaling Laws, Fine-Tuning and In-Context Learning →

Problem

Justify $C\approx6ND$ for training a dense transformer, then estimate the training compute for $N = 7\times10^9$ parameters and $D = 2\times10^{12}$ tokens, and convert to GPU-days at $4\times10^{14}$ effective FLOP/s per device.

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Count the FLOPs of a forward pass	LLM Ops	Numerical	Foundation	15 min
Derive the attention crossover length	LLM Ops	Proof	Beginner	15 min
LoRA parameter accounting	LLMs	Numerical	Beginner	15 min
When does attention dominate the MLP?	Deep Learning	Numerical	Beginner	30 min
Derive the compute-optimal allocation	LLMs	Proof	Intermediate	30 min