Sizing a Serving Cluster End to End

The three constraints

Fix a replica and a request profile. Write:

• $M_{\text{kv}}$ — bytes of device memory left for KV cache after weights and workspace;
• $m$ — KV bytes for one resident sequence at its expected length;
• $\eta \in (0,1]$ — the fraction of the cache actually usable, after fragmentation;
• $W$ — mean end-to-end request latency, the quantity the SLO constrains;
• $\lambda$ — offered load in requests per second for the whole service;
• $\rho^\star$ — the target utilisation, chosen from the utilisation wall.

PropositionThe memory ceiling on concurrency

The number of requests a replica can hold concurrently is at most

$$L_{\max} \;=\; \left\lfloor \frac{\eta\,M_{\text{kv}}}{m} \right\rfloor .$$

ProofReveal

PropositionMaximum throughput of one replica

$$X_{\text{replica}} \;\le\; \frac{L_{\max}}{W} .$$

ProofReveal

TheoremThe sizing formula

To serve offered load $\lambda$ at mean latency $W$ while keeping each replica at utilisation $\rho^\star$, the number of replicas required is

$$\boxed{\;R \;=\; \left\lceil \frac{\lambda\,W}{\rho^\star\,L_{\max}} \right\rceil \;=\; \left\lceil \frac{\lambda\,W\,m}{\rho^\star\,\eta\,M_{\text{kv}}} \right\rceil \;}$$

ProofReveal

Working an example

Take a replica with $M_{\text{kv}} = 40$ GB of spare memory, a request profile whose expected KV footprint is $m = 500$ MB, and paged allocation giving $\eta = 0.9$. Then

$$L_{\max} \;=\; \left\lfloor \frac{0.9 \times 40\,\text{GB}}{0.5\,\text{GB}} \right\rfloor \;=\; 72 .$$

With mean latency $W = 4$ s and target utilisation $\rho^\star = 0.7$, one replica sustains $0.7 \times 72/4 = 12.6$ requests per second. Serving $\lambda = 500$ requests per second needs

$$R \;=\; \left\lceil \frac{500 \times 4}{0.7 \times 72} \right\rceil \;=\; \lceil 39.7 \rceil \;=\; 40 \ \text{replicas} .$$

Now change one thing. Halving $m$ — by quantising the KV cache to 8 bits — doubles $L_{\max}$ to $144$ and halves $R$ to $20$. Halving $W$ does the same. Raising $\rho^\star$ from $0.7$ to $0.9$ reduces $R$ to $31$ if $W$ stayed fixed, but $W$ does not stay fixed: by the utilisation wall the queueing component grows by a factor $(0.9/0.1)/(0.7/0.3) = 3.86$, and unless queueing was a small part of $W$ to begin with, the change is a net loss.

Which constraint binds

CorollaryWhich constraint binds

There is a second ceiling on concurrency, from compute: the batch size at which decode stops being weight-bound. Call it $L_{\text{compute}}$. The replica's effective cap is $\min(L_{\max}, L_{\text{compute}})$, and:

• if $L_{\max} < L_{\text{compute}}$ the replica is memory-bound — buy KV capacity, not FLOPs;
• if $L_{\text{compute}} < L_{\max}$ it is compute-bound — a bigger cache buys nothing.

ProofReveal

By the decode arithmetic in Inference Arithmetic and the Roofline, arithmetic intensity in weight-limited decoding is $2B/b$ for batch $B$ and $b$ bytes per parameter, so $L_{\text{compute}}$ is roughly the batch at which $2B/b$ reaches the hardware's compute-to-bandwidth ratio. For a machine with ratio $\approx 300$ and $b = 2$, that is $B \approx 300$. A replica whose memory allows only $72$ concurrent sequences is therefore firmly memory-bound, and its idle arithmetic units are not a bug to be tuned away — they are the direct consequence of the KV budget.