Autoscaling — Delay, Overshoot and Standing Headroom

Why a scaling loop with a delay must keep spare capacity it never uses, and exactly how long a system stays behind after a load step.

AI SystemsGraduate⏱ 1 hour6 min readautoscalingcontroldelayovershootheadroom

AssumesQueueing and the Utilisation Wall · Sizing a Serving Cluster End to End

What the delay costs

PropositionBacklog accumulated during the scaling delay

Suppose the system has capacity $\mu$ and offered load steps from $\lambda_0 < \mu$ to $\lambda_1 > \mu$ at time $0$ , and that new capacity becomes available only at time $d$ . Then the backlog at time $d$ is

Q(d) \;=\; (\lambda_1-\mu)\,d ,

independent of the scaling policy.

How long the system stays behind

TheoremRecovery time after a load step

With capacity raised to $\mu' > \lambda_1$ at time $d$ , the backlog clears at

T_{\text{recover}} \;=\; d \;+\; \frac{Q(d)}{\mu'-\lambda_1} \;=\; d\,\cdot\,\frac{\mu'-\mu}{\mu'-\lambda_1} .

Key idea

Read the formula as a warning about timid scaling. Scaling to $\mu'$ only just above $\lambda_1$ makes the denominator small and the recovery time blow up: provisioning 10% above the new load when the step was 50% gives $T_{\text{recover}} = d\,(1.5\cdot 1.1 - 1)/(1.65-1.5)$ — with $\mu=1$ , $\lambda_1 = 1.5$ , $\mu' = 1.65$ — which is $d \times 0.65/0.15 \approx 4.3\,d$ . Overshooting to $\mu' = 3$ gives $d \times 2/1.5 \approx 1.3\,d$ . Deliberate overshoot is how a scaler recovers quickly, and it is cheap because it is transient.

The headroom you must always carry

CorollaryStanding headroom needed to absorb a step

To absorb an instantaneous load step by a factor $g > 1$ with no backlog at all, the pre-step utilisation must satisfy

\rho_0 \;=\; \frac{\lambda_0}{\mu} \;\le\; \frac 1g .

So absorbing a doubling requires running at 50%; absorbing a 4× spike requires 25%. Combined with the utilisation wall — which already forbids running near 1 — this is the quantitative case against "scale to zero and react", and it explains why serving fleets look over-provisioned when measured by average utilisation alone.

HeuristicScale on queueing delay rather than utilisation

Heuristic — a reasoning shortcut that predicts the right answer in the regimes discussed here. It is not a proof and can fail outside them.

Utilisation is bounded above by $1$ and saturates: at $\rho = 0.95$ and $\rho = 0.999$ it reads almost the same, while latency differs by a factor of $20$ . Queue depth and queueing delay are unbounded and grow monotonically with excess load, so they distinguish the two. A controller that targets queueing delay therefore has a usable error signal in exactly the regime where a utilisation-based controller has none.

Assumptions

This is an argument from the shape of the signals, not a theorem about a specific controller. Its force depends on the delay measurement being timely: a queueing-delay estimate averaged over a window longer than $d$ reintroduces the blindness it was meant to remove.

What this does not say

This page models capacity as a number that changes at time $d$ . For model serving it is not: a new replica must load tens of gigabytes of weights and warm its caches before it is useful, so there is a further interval during which the replica exists, consumes quota, and serves nothing — and if the load balancer routes to it on readiness rather than warmth, it will serve that traffic slowly. Weight-loading time is the dominant term in $d$ for LLM serving, which is why snapshotting and pre-warming pools matter more here than in a stateless web tier.

References

PaperA. Gandhi, M. Harchol-Balter, R. Raghunathan, M. A. Kozuch (2012). AutoScale: Dynamic, Robust Capacity Management for Multi-Tier Data CentersStudies the setup-delay problem and argues for keeping idle capacity, consistent with the headroom corollary.

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

PropositionBacklog accumulated during the scaling delayproof includedA load step above capacity builds a backlog equal to the excess rate times the delay, whatever the controller does afterwards.
TheoremRecovery time after a load stepproof includedTime to clear the backlog is the delay multiplied by the ratio of total capacity added to the surplus over the new load.
CorollaryStanding headroom needed to absorb a stepproof includedTo absorb a load step of factor g without any backlog, steady-state utilisation must not exceed 1/g.
HeuristicScale on queueing delay rather than utilisationUtilisation is a bounded signal that saturates before the SLO breaks, while queue depth keeps growing, making it the better controller input.