Count the FLOPs of a forward pass

NumericalFoundation⏱ 15 minLLM Opsflopscountingprefill

Problem

A decoder-only transformer has $N = 6.7\times10^9$ non-embedding parameters, $L = 32$ layers and width $d = 4096$ .

Compute the FLOPs of a forward pass over a 512-token prompt.
Compute it over a 32,768-token prompt.
In each case, give the fraction contributed by the attention score/value products.

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
Adapter parameter counts	Fine-Tuning	Numerical	Foundation	15 min
Derive the attention crossover length	LLM Ops	Proof	Beginner	15 min
Ridge point and the batch size needed to saturate	LLM Ops	Numerical	Beginner	15 min
The 6ND FLOP estimate	LLMs	Numerical	Beginner	15 min
When does attention dominate the MLP?	Deep Learning	Numerical	Beginner	30 min