Autoscaling and Cold Starts

Why a GPU replica takes minutes to become useful, what that does to a scaling policy, and the four techniques that shorten it.

AI EngineeringAdvanced⏱ 3 hours7 min readautoscalingcold startwarm poolscale to zeromodel loading

AssumesGPU Routing and Cache Affinity · Autoscaling — Delay, Overshoot and Standing Headroom

The budget

DefinitionWhat the cold start is made of

Stage	Typical	Attack
scheduling onto a node with a free GPU	5–60 s	pre-provisioned node pool, or reservations
container image pull	30–300 s	slim images, layer caching, pre-pulled on the node
weight load into device memory	60–600 s	local NVMe cache, tensor streaming, snapshots
CUDA context + kernel compilation	10–90 s	AOT compilation, cached compilation artefacts
warm-up generations	5–30 s	run before joining the router, not after
total	2–10 min

The dominant term is almost always the weights, and it has a hard floor.

PropositionA lower bound on weight loading

A model of $N$ parameters at $b$ bytes each cannot be loaded into device memory faster than

t_{\text{load}} \;\ge\; \frac{N b}{B_{\text{eff}}} ,

where $B_{\text{eff}}$ is the narrowest bandwidth on the path from where the weights are stored to device memory.

Key idea

Put numbers on it. A 70-billion-parameter model in bf16 is $140$ GB.

Path	Bandwidth	Floor
object storage over a shared network	1 GB/s	140 s
local NVMe	5 GB/s	28 s
host RAM → device over PCIe 4 ×16	~25 GB/s	6 s

The remedy is visible directly in the table: move the weights closer before you need them. A node-local NVMe cache turns a 140-second load into 28; keeping the weights resident in host RAM turns it into 6. Both are pre-work, and both are what "warm pool" actually means in practice.

Should you scale to zero?

PropositionWhen scale-to-zero pays

Over a period $T$ , let $f$ be the fraction of time no request is in flight, $n$ the number of idle-to-active transitions, $d$ the cold start and $c$ the cost per replica-second. Keeping one replica warm costs $cT$ . Scaling to zero costs

c\,(1-f)\,T \;+\; c\,n\,d ,

since each cold start burns $d$ replica-seconds before serving anything. Scale-to-zero is cheaper exactly when

\boxed{\ f\,T \;>\; n\,d\ }

— the total idle time must exceed the total time spent starting up. Separately, $n$ requests per period wait $d$ for their first token, which is a latency cost the inequality does not price.

Key idea

Put numbers in. With $d = 5$ minutes and a service idle 20 hours a day, scale-to-zero pays while $n < fT/d = (20 \times 60)/5 = 240$ transitions a day. That sounds generous until you notice that a service probed by a health checker every minute never goes idle at all, and one used sporadically by twenty people can easily transition a few hundred times.

So the rule is: scale to zero when the traffic is clustered — long genuine idle periods, few transitions — and never when it is sparse but spread out, which is the pattern that looks idle on a utilisation graph and thrashes on a cold-start graph. Interactive diurnal traffic is the worst case: one transition, at exactly the moment every user arrives, all of them paying $d$ together.

Scaling that arrives in time

HeuristicScale on queued work, with a lead

Heuristic — a reasoning shortcut that predicts the right answer in the regimes discussed here. It is not a proof and can fail outside them.

Two properties the signal and the policy must have:

The signal must not saturate. GPU utilisation reads near 100% at both "comfortably busy" and "catastrophically overloaded", so it cannot distinguish them. Queued tokens, queue depth and time-to-first-token all keep growing past saturation and can.

The policy must lead the traffic by $d$ . A reactive controller starts the replica when the queue is already long, and by the recovery-time result the system stays behind for $d(\mu'-\mu)/(\mu'-\lambda_1)$ . Leading requires prediction: schedule-based (traffic is diurnal and you know the shape), derivative-based (the queue is growing at a rate that will breach in $d$ seconds), or simply a standing headroom of $1/g$ from the step-absorption corollary.

Assumptions

Prediction is only as good as the regularity of the traffic. For a genuinely unpredictable spike — a viral post, a batch job someone launched — no lead time is available, and the only defences are standing headroom and graceful degradation. The prediction is for the 95% of load that is predictable; the headroom is for the rest.

def desired_replicas(now, metrics, policy):
    # reactive term: what the current queue needs
    need_now = ceil(metrics.queued_tokens / policy.tokens_per_replica_per_sec / policy.target_ttft)

    # predictive term: what we will need once a new replica is ready
    growth = metrics.queued_tokens_derivative                 # tokens/s^2
    need_then = ceil((metrics.queued_tokens + growth * policy.cold_start_seconds)
                     / policy.tokens_per_replica_per_sec / policy.target_ttft)

    # schedule term: known diurnal shape, if we have one
    need_sched = policy.forecast(now + policy.cold_start_seconds)

    target = max(need_now, need_then, need_sched, policy.floor)
    return min(target, policy.ceiling)

The four techniques

Warm pools — keep $k$ replicas loaded and idle, sized from the step-absorption bound. The simplest and most reliable; it costs exactly what it looks like it costs.
Node-local weight cache — keep checkpoints on NVMe so a restart reads at 5 GB/s rather than pulling from object storage. Cheap, and the largest single win in the table above.
Snapshot and restore — checkpoint the initialised process, including device memory, and restore it. Skips CUDA init and compilation as well as the load.
Over-provision during known windows — if traffic is diurnal, scale on the clock and let the reactive controller handle only the residual.

What this does not say

Cold starts interact with the adapter routing in GPU Routing and Cache Affinity: a fleet serving many per-tenant adapters has a second, much smaller cold start per adapter, and a scaling event throws away every resident adapter along with the KV cache. Systems that scale frequently and multiplex adapters heavily pay this repeatedly, and it is invisible in a cold-start metric that measures only the base model.

References

PaperY. Fu et al. (2024). ServerlessLLM: Low-Latency Serverless Inference for Large Language ModelsMeasures the checkpoint-loading term and attacks it with multi-tier caching and live migration.
PaperA. Gandhi, M. Harchol-Balter, R. Raghunathan, M. A. Kozuch (2012). AutoScale: Dynamic, Robust Capacity Management for Multi-Tier Data CentersThe setup-delay analysis and the argument for keeping idle capacity.

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

DefinitionWhat the cold start is made ofScheduling, image pull, weight load, CUDA initialisation, compilation and warm-up, each with a different remedy.
PropositionA lower bound on weight loadingproof includedLoading N parameters at b bytes each over a link of bandwidth B cannot take less than Nb/B seconds, which for a large model over object storage is minutes.
PropositionWhen scale-to-zero paysproof includedScaling to zero is cheaper than a warm replica only when the idle fraction exceeds the cold-start cost amortised over the requests it delays.
HeuristicScale on queued work, with a leadThe controller must act on a signal that grows past saturation and must act early enough to cover the cold start, which means predicting rather than reacting.