Skip to content
nonconvexcard

The Economics of Serving — Cost per Token

Deriving unit cost from hardware price, throughput and utilisation, and the exact price of a tighter latency target.

AI SystemsAdvanced⏱ 1 hour6 min readcostunit economicsutilisationlatencypricing
1 problem →
AssumesSizing a Serving Cluster End to End · Queueing and the Utilisation Wall

The unit cost

PropositionCost per token

Let a replica cost cc per hour, have maximum sustainable output Xmax⁡X_{\max} tokens per second, and run at utilisation ρ\rho. Then

cost per token  =  c3600  ρ Xmax⁡.\text{cost per token} \;=\; \frac{c}{3600\;\rho\,X_{\max}} .

Key idea

Three multiplicative factors, and only one of them is a procurement decision:

cost per token  =  c⏟what you pay×1Xmax⁡⏟how fast it can go×1ρ⏟how much you use it×13600\text{cost per token} \;=\; \underbrace{c}_{\text{what you pay}} \times \underbrace{\frac{1}{X_{\max}}}_{\text{how fast it can go}} \times \underbrace{\frac{1}{\rho}}_{\text{how much you use it}} \times \frac{1}{3600}

A replica at 30% utilisation costs 3.3×3.3\times per token what the same replica costs at 100%. Negotiating 20% off the hourly rate is worth less than moving utilisation from 50% to 65%.

What a latency target costs

Utilisation cannot be pushed freely, because latency rises with it. Making that explicit turns the cost model into a statement about the latency target itself.

TheoremThe cost of a latency target

Model a replica as a single-server queue with mean service time SS and mean latency W=S/(1−ρ)W = S/(1-\rho). Then achieving a mean-latency target WW requires ρ=1−S/W\rho = 1 - S/W, and

cost per token  =  c3600 Xmax⁡⋅WW−S.\text{cost per token} \;=\; \frac{c}{3600\,X_{\max}}\cdot\frac{W}{W-S} .

The factor W/(W−S)W/(W-S) is decreasing in WW, equals 11 in the limit of an unbounded latency budget, and diverges as W↓SW \downarrow S.

Assumptions

W=S/(1−ρ)W = S/(1-\rho) is the M/M/1 relation from Queueing and the Utilisation Wall. A different service distribution changes the constant through the variability factor (1+cs2)/2(1+c_s^2)/2 but not the shape: every work-conserving queue has a 1/(1−ρ)1/(1-\rho) term, so every such system has a cost that diverges as the latency target approaches the bare service time.

Reading off the exchange rate, with SS as the unit:

latency target WW2S2S3S3S4S4S10S10S∞\infty
utilisation ρ\rho0.500.670.750.901
relative cost2.001.501.331.111

Key idea

Tightening the mean-latency target from 4S4S to 2S2S raises unit cost by 2.00/1.33=1.52.00/1.33 = 1.5 — 50% more expensive, for a 2×2\times latency improvement. That is the whole trade, in one number, and it is why interactive and batch traffic belong in separate pools: batch work has W=∞W = \infty and should be run at the utilisation that implies, subsidising nothing and being subsidised by nothing.

Where Xmax⁡X_{\max} comes from

CorollaryWhy batching dominates the cost structure

In weight-bound decoding, each decode step streams the weights once regardless of batch size, so a step takes time ≈Nb/Bmem\approx N b / B_{\text{mem}} and emits BB tokens. Hence

Xmax⁡  ≈  B BmemN b,X_{\max} \;\approx\; \frac{B\,B_{\text{mem}}}{N\,b},

linear in the batch BB, and unit cost falls like 1/B1/B until the arithmetic units saturate.

Key idea

Unit cost is inversely proportional to batch size, and by Sizing a Serving Cluster End to End the achievable batch is Lmax⁡=ηMkv/mL_{\max} = \eta M_{\text{kv}}/m — a memory quantity. So the dominant lever on cost per token is KV-cache footprint, not FLOPs. Halving mm through grouped-query attention or KV quantisation halves unit cost in the memory-bound regime, which no amount of arithmetic throughput would have done.

What this does not say

This is a model of marginal serving cost. It excludes training amortisation, which for a frequently-retrained model can exceed serving cost entirely; it excludes the cost of the over-provisioned headroom that Autoscaling — Delay, Overshoot and Standing Headroom shows is mandatory; and it assumes the replica is billed continuously, which is wrong for spot or serverless capacity where the pricing model interacts with the scaling delay. Each of those is a real term, and each moves the answer in the same direction.

References

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

  • PropositionCost per tokenproof includedUnit cost is the hourly price of a replica divided by the tokens it actually delivers, which is its capacity scaled by utilisation.
  • TheoremThe cost of a latency targetproof includedUnder a single-server queueing model, cost per token is proportional to W/(W−S), so it diverges as the latency target approaches the bare service time.
  • CorollaryWhy batching dominates the cost structureproof includedIn the weight-bound decode regime, tokens per second scale with batch size, so unit cost falls roughly inversely with the batch a replica can hold.

Problems (1)

All problems