Apply Little's law to a serving target
ProofIntermediate⏱ 30 minLLM Opslittle's lawcapacityslo
Problem
A service must sustain 40 requests per second. Each request averages 800 output tokens, and the product requires a mean end-to-end latency of 20 seconds.
- Use Little's law to find the required mean concurrency.
- Given the capacity table from the previous problem at 8k context, how many GPUs are needed?
- Show that if latency is halved, the required concurrency halves — and explain why this does not halve the GPU count.
Write a complete argument. Every inequality you use should be one you could state precisely.
Solution
Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.
Related problems
All problems| Solved | Problem | Field | Type | Difficulty | Time |
|---|---|---|---|---|---|
| KV cache sizing with MHA and GQA | LLM Ops | Numerical | 30 min | ||
| How many users fit on one GPU | LLM Ops | Numerical | 30 min | ||
| Prove Little's law on a finite horizon | AI Systems | Proof | 1 hour | ||
| Recover Little's law from an event log run | AI Systems | Coding | 1 hour | ||
| Simulate M/M/1 and verify the utilisation wall run | AI Systems | Coding | 1 hour |