Derive the attention crossover length

ProofBeginner⏱ 15 minLLM Opscrossoverflopsscaling

Theory: Inference Arithmetic and the Roofline →

Problem

Show that the attention term of the forward-pass cost overtakes the weight term at $S_\star = N/(2Ld)$ . Then show that for a transformer whose parameters are dominated by the standard per-layer blocks, $S_\star$ is proportional to $d$ , and say what that implies about wide versus deep models at long context.

Write a complete argument. Every inequality you use should be one you could state precisely.

Hints

Hints — 0 of 2 revealed

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Count the FLOPs of a forward pass	LLM Ops	Numerical	Foundation	15 min
Ridge point and the batch size needed to saturate	LLM Ops	Numerical	Beginner	15 min
The 6ND FLOP estimate	LLMs	Numerical	Beginner	15 min
When does attention dominate the MLP?	Deep Learning	Numerical	Beginner	30 min
Locating the Bernstein crossover	High-Dim Probability	Numerical	Intermediate	30 min