Deployment and Rollout

Shipping a model change: the registry, shadow traffic, canaries, and why a config change deserves more caution than a code change.

AI EngineeringAdvanced⏱ 1 hour4 min readdeploymentcanaryshadow trafficmodel registryrollback

AssumesEvaluation Harnesses · Rollout, Canaries and Online Experiments

DefinitionSix things whose change alters behaviour

Artefact	Usual rigour	Deserved rigour
service code	review, CI, canary	✓
model version	often an alias	pin, eval, canary
system prompt	edited in a config file	review, eval, canary
tool schemas	shipped with code	✓, plus a compatibility check
retrieval index	rebuilt on a schedule	versioned, eval, atomic swap
sampling parameters	a dashboard field	reviewed like code
guardrail config	a dashboard field	reviewed like code

Everything in the table should go through the same pipeline and appear in the same change log. The failure that takes longest to diagnose is always the one that did not look like a deployment.

DefinitionThe rollout ladder

Stage	Traffic	Answers	Catches
offline eval	none	is it better on what we can measure?	regressions you anticipated
shadow	100%, responses discarded	does it crash, and how fast is it?	latency, errors, cost — at full scale, zero user risk
canary	1–5%, real responses	do users react differently?	quality regressions your evals missed
progressive	5 → 25 → 50 → 100%	does it hold at scale?	capacity and long-tail effects

Shadow is the step most often skipped and the cheapest insurance: it exercises the new path against real production traffic distribution with no possibility of user harm. It cannot tell you anything about quality, because nobody sees the output — that is what the canary is for.

PropositionA config change reaches every replica at once

A rolling code deploy updates replicas in batches, so a bad build is exposed to a fraction of traffic before it is halted. Configuration distributed by a control plane is fetched by all replicas within one refresh interval. In the beta-factor model of Availability, Redundancy and Correlated Failure this is $\beta = 1$ : every replica fails together, and the achievable unavailability floor $\beta U$ equals the unavailability of a single replica. Redundancy provides no protection at all against it.

Key idea

Configuration must therefore be rolled out like code — staged by percentage of replicas or traffic, with the same halt-on-error automation — or the redundancy the fleet was built for is inoperative for that class of change. "It's just a config change" is precisely the reasoning that produces the worst outages, and the mathematics says why.

HeuristicBuild the rollback before the rollout

Heuristic — a reasoning shortcut that predicts the right answer in the regimes discussed here. It is not a proof and can fail outside them.

Before shipping, three questions must have answers:

What is the revert action, exactly? A pointer swap, a flag flip, a redeploy of a pinned version — named, and tested in staging.
How long does it take? If reverting a model change means a cold start on every replica, the rollback takes as long as the cold start, and that is your worst-case exposure.
Does it depend on the thing being reverted? A rollback that needs the control plane, when the bad config broke the control plane, is not a rollback.

DEPLOY = {"model": "claude-opus-4-5-20251101",
          "prompt_version": "support-v14",
          "index_version": 41,
          "previous": {"model": "claude-opus-4-5-20251015",
                       "prompt_version": "support-v13", "index_version": 40}}

Carrying the previous configuration in the current one makes reverting a single atomic write rather than an archaeology exercise during an incident.

What this does not say

Rolling back a model does not roll back its effects. Conversations already stored, tool calls already executed, documents already written and caches already populated persist across the revert — and a semantic cache filled by the bad version will keep serving its answers afterwards. Cache invalidation on rollback is part of the revert action, and is routinely forgotten.

References

BookB. Beyer, C. Jones, J. Petoff, N. R. Murphy (2016). Site Reliability Engineering: How Google Runs Production SystemsProgressive rollout and the canarying practice followed here.

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

DefinitionSix things whose change alters behaviourModel version, system prompt, tool schemas, retrieval index, sampling parameters and guardrail configuration are all deployments.
DefinitionThe rollout ladderOffline eval, then shadow, then canary, then progressive rollout, each answering a question the previous one could not.
HeuristicBuild the rollback before the rolloutA change is only safe to ship if reverting it is a single fast action that does not depend on the thing being reverted.
PropositionA config change reaches every replica at onceproof includedConfiguration distributed by a control plane propagates without a rolling deploy, so it is a beta-equal-one correlated failure in the redundancy model.