Price a latency target

NumericalIntermediate⏱ 30 minAI Systemscostlatencyutilisationtradeoff

Problem

A replica costs $2.80 per hour and can emit 4000 tokens per second at saturation. Its bare mean service time is $S$ .

Derive cost per million tokens as a function of the mean-latency target $W$ , in units of $S$ .
Tabulate it for $W/S \in \{1.5, 2, 4, 10\}$ .
A product team asks to move from $W = 4S$ to $W = 1.5S$ . By what factor does unit cost rise?
Batch and interactive traffic currently share one pool at $W = 2S$ . Half the tokens are batch and could tolerate $W = 10S$ . What is the blended saving from splitting the pools?

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
Sizing best-of-n	Agents	Numerical	Beginner	15 min
Derive the M/M/1 queue from detailed balance	AI Systems	Proof	Intermediate	1 hour
Simulate M/M/1 and verify the utilisation wall run	AI Systems	Coding	Intermediate	1 hour
Single-stream decode latency floors	LLM Ops	Numerical	Intermediate	30 min
Size a serving cluster and find the binding constraint	AI Systems	Numerical	Intermediate	1 hour