AI System Design
The mathematics of putting a model into production: Little’s law and the utilisation wall, tail latency under fan-out, parallelism and collective-communication cost models, caching, load balancing, availability and cost per token.
Level 1 — Core
- What a Serving System Must Guarantee9 min
The vocabulary the rest of this field is written in — offered load, utilisation, throughput, concurrency and latency percentiles — defined precisely enough to compute with.
Stability requires utilisation below oneThe mean constrains the tail only by a factor of 1/(1-p)Percentiles do not add - Little's Law and the Capacity of a Serving System11 min
The one identity that holds for any queueing system whatsoever — L = λW — proved from an area argument, and the sizing rules that follow from it.
Little's lawThe utilisation lawConcurrency needed for a throughput target
Level 2 — Advanced
- Queueing and the Utilisation Wall13 min
Why latency diverges like 1/(1−ρ) as a server approaches saturation, derived exactly for M/M/1 and M/G/1, and why variance costs as much as load.
Stationary distribution of M/M/1Mean latency of M/M/1The Pollaczek–Khinchine mean formula - Tail Latency and Fan-Out Amplification8 min
Why a request that touches a hundred machines is slow even when every machine is fast, and what hedging buys — derived from the distribution of a maximum.
The distribution of the slowest of n callsFan-out shifts which quantile you must controlA hedged request truncates the tail at the hedge delay - Scaling Limits — Amdahl, Gustafson and the Coherency Penalty8 min
Three models of what happens when you add machines: a hard ceiling, an escape from it, and the case where throughput actually falls.
Amdahl's lawGustafson's scaled speedupThe universal scalability law has an interior maximum - Sizing a Serving Cluster End to End8 min
Putting Little's law, the KV-cache budget and the utilisation wall together into a single sizing formula, and reading off which constraint actually binds.
The memory ceiling on concurrencyMaximum throughput of one replicaThe sizing formula - Caching — Hit Rates, Zipf and the Working Set9 min
What a cache of a given size actually buys under realistic popularity skew, and why prefix caching is governed by a different quantity than object caching.
Effective latency under a cacheStatic optimal hit rate under a Zipf popularity lawThe power-law form of the hit rate - Load Balancing and the Power of Two Choices7 min
Why sampling two servers and taking the shorter queue is exponentially better than sampling one, and why least-outstanding-requests beats round robin for variable work.
Maximum load under uniform random assignmentThe power of two choicesA log n / log log n upper bound for one choice - Availability, Redundancy and Correlated Failure7 min
The arithmetic of nines — why chains multiply, why replicas help less than they look like they should, and the floor that correlated failure puts under any amount of redundancy.
Availability of a dependency chainAvailability of independent replicasQuorum availability - The Economics of Serving — Cost per Token6 min
Deriving unit cost from hardware price, throughput and utilisation, and the exact price of a tighter latency target.
Cost per tokenThe cost of a latency targetWhy batching dominates the cost structure
Level 3 — Graduate
- The Cost of Collective Communication7 min
The alpha–beta model, and why a ring all-reduce moves a bounded amount of data per node no matter how many nodes there are.
The alpha–beta cost modelCost of a centralised all-reduceCost of the ring all-reduce - Splitting a Model Across Devices9 min
Data, tensor and pipeline parallelism compared by the one quantity that decides between them — bytes communicated per FLOP of useful work.
Data-parallel communication does not depend on model sizeTensor-parallel communication does not depend on batch sizeThe pipeline bubble fraction - Scheduling, Admission Control and Fairness8 min
Why shortest-remaining-first minimises mean latency, what it costs the long requests, and why a bounded queue is a feature rather than a limitation.
SRPT minimises mean response timeWhat SRPT costs the largest jobsHead-of-line blocking under FCFS - Autoscaling — Delay, Overshoot and Standing Headroom6 min
Why a scaling loop with a delay must keep spare capacity it never uses, and exactly how long a system stays behind after a load step.
Backlog accumulated during the scaling delayRecovery time after a load stepStanding headroom needed to absorb a step - Rollout, Canaries and Online Experiments8 min
How much traffic a canary needs before it can detect a regression, why stopping when the result looks good inflates the error rate, and what a guardrail suite costs in sensitivity.
Sample size for a two-arm comparisonHow long a canary must runRepeated testing inflates the false-positive rate
Problems (16)
Open in the browser| Solved | Problem | Type | Difficulty | Time |
|---|---|---|---|---|
| Budget a latency target across a fan-out | Numerical | 15 min | ||
| Implement the cluster sizing formula run | Coding | 30 min | ||
| Balls into bins, with one choice and with two run | Coding | 1 hour | ||
| Cache hit rates under Zipf popularity run | Coding | 1 hour | ||
| Derive the M/M/1 queue from detailed balance | Proof | 1 hour | ||
| How long must the canary run? | Numerical | 30 min | ||
| Measure fan-out amplification and the value of hedging run | Coding | 1 hour | ||
| Price a latency target | Numerical | 30 min | ||
| Prove Little's law on a finite horizon | Proof | 1 hour | ||
| Recover Little's law from an event log run | Coding | 1 hour | ||
| Simulate M/M/1 and verify the utilisation wall run | Coding | 1 hour | ||
| Size a serving cluster and find the binding constraint | Numerical | 1 hour |