Inference Arithmetic and the Roofline

Counting the work

Fix a decoder-only transformer with $L$ layers, model width $d$, and $N$ non-embedding parameters. We count multiply–accumulate as two FLOPs, the standard convention.

PropositionForward-pass cost of a transformer

A forward pass over a sequence of $S$ tokens costs

$$C_{\text{fwd}}(S) \ \approx\ \underbrace{2NS}_{\text{weights}} \;+\; \underbrace{4LdS^2}_{\text{attention}} \quad\text{FLOPs}.$$

ProofReveal

When does attention actually dominate?

The quadratic term overtakes the linear one when $4LdS^2 > 2NS$, that is when

$$S \;>\; S_\star \;:=\; \frac{N}{2Ld}.$$

For a 7B-parameter model with $L = 32$ and $d = 4096$:

$$S_\star = \frac{6.7\times 10^9}{2 \cdot 32 \cdot 4096} \approx 2.6 \times 10^4 .$$

The roofline

PropositionThe roofline bound

Let a computation perform $F$ FLOPs while moving $Q$ bytes between memory and the processor, and define its arithmetic intensity $I = F/Q$. On hardware with peak throughput $\pi$ FLOP/s and memory bandwidth $\beta$ byte/s, the attainable throughput satisfies

$$P \ \le\ \min\big(\pi,\ I\beta\big).$$

ProofReveal

The two regimes meet at the ridge point $I^\star = \pi/\beta$. A kernel with $I < I^\star$ is memory-bound; one with $I > I^\star$ is compute-bound.

The two phases have opposite intensity

Prefill

Processing a prompt of $S$ tokens does $\approx 2NS$ FLOPs while loading the $N$ weights once, i.e. $Nb$ bytes for $b$ bytes per parameter. Its intensity is

$$I_{\text{prefill}} \approx \frac{2NS}{Nb} = \frac{2S}{b}.$$

At $b = 2$ (BF16), $I_{\text{prefill}} \approx S$. Any prompt longer than a few hundred tokens clears the ridge point comfortably: prefill is compute-bound.

Decode

PropositionDecode arithmetic intensity is set by the batch size alone

Generating one token for each of $B$ sequences, in the regime where weight traffic dominates, has arithmetic intensity

$$I_{\text{decode}} \approx \frac{2B}{b},$$

independent of $N$, $L$ and $d$.

ProofReveal

At $b = 2$, $I_{\text{decode}} \approx B$. To reach an H100's ridge point you need a batch of roughly 300 concurrent sequences. Below that, the tensor cores are idle most of the time and the only thing that matters is how fast weights stream out of HBM.

CorollaryA hard floor on single-stream decode latency

With $N$ parameters at $b$ bytes each and memory bandwidth $\beta$, the time to emit one token satisfies

$$t_{\text{step}} \ \ge\ \frac{Nb}{\beta},$$

whatever the implementation.

ProofReveal

Worked example: the ceiling on single-stream generation

What this predicts, and where it fails

Empirical observation

Empirical observation — a regularity seen in experiments. No theorem on this page establishes it in general.

Measured decode throughput on production stacks rises roughly linearly with batch size in the small-batch regime and then flattens, with the knee in the low hundreds of sequences — the qualitative shape this analysis predicts.

Open problem

Open problem — no proof and no counterexample is known to the author of this page.

There is no accepted analytical model that predicts achieved throughput of a full serving stack from model shape and hardware specifications. Practitioners benchmark. The roofline gives a ceiling and the gap to it is measured, not derived — a situation that is normal in performance engineering and worth stating plainly rather than papering over.