Rollout, Canaries and Online Experiments

How much traffic a canary needs before it can detect a regression, why stopping when the result looks good inflates the error rate, and what a guardrail suite costs in sensitivity.

AI SystemsGraduate⏱ 3 hours8 min readA/B testingcanarysequential testingpowerguardrails

1 problem →

AssumesAvailability, Redundancy and Correlated Failure · Concentration Inequalities

How much traffic you need

PropositionSample size for a two-arm comparison

To detect a true difference of $\Delta$ in the mean of a metric with per-observation variance $\sigma^2$ in each arm, using a two-sided test at level $\alpha$ with power $1-\beta$ , the required sample size per arm is

n \;=\; \frac{2\sigma^2\,\bigl(z_{1-\alpha/2} + z_{1-\beta}\bigr)^2}{\Delta^2} .

For $\alpha = 0.05$ and $1-\beta = 0.8$ the constant is $2(1.96+0.84)^2 = 15.7$ , so

n \;\approx\; \frac{16\,\sigma^2}{\Delta^2} .

Assumptions

Observations are independent within and across arms. Correlated units — repeated requests from the same user, or requests sharing a replica — inflate the effective variance, and the standard fix is to randomise and analyse at the unit of independence rather than per request.
The difference-in-means estimator is approximately normal, which for the sample sizes below follows from the central limit theorem.

CorollaryHow long a canary must run

If total traffic is $\lambda$ requests per second and the canary receives a fraction $f$ , then both arms can be sized at $n$ only after

T \;=\; \frac{n}{f\,\lambda} \;\approx\; \frac{16\,\sigma^2}{f\,\lambda\,\Delta^2}

seconds, since the canary arm is the scarce one.

Key idea

Work an example. Suppose you want to detect a 1% relative regression in a latency metric whose coefficient of variation is $1$ — so $\sigma \approx$ the mean $m$ , and $\Delta = 0.01m$ . Then

n \;\approx\; \frac{16\,m^2}{(0.01m)^2} \;=\; 16\times10^4 \;=\; 160{,}000 \ \text{per arm}.

At 1000 requests per second with a 1% canary, $f\lambda = 10$ per second, so $T = 16{,}000$ s $\approx 4.4$ hours. A 1% canary cannot detect a 1% regression quickly, and the quadratic dependence on $\Delta$ is unforgiving: detecting 0.5% instead takes four times as long.

Why watching continuously breaks the test

PropositionRepeated testing inflates the false-positive rate

Suppose the null hypothesis holds and you run a fixed-sample test at level $\alpha$ at each of $k$ interim times, stopping and declaring significance the first time it rejects. The probability of declaring significance at some point is

\alpha_k \;=\; \Pr\left[\bigcup_{j=1}^k R_j\right] \;\ge\; \alpha,

strictly increasing in $k$ , and $\alpha_k \to 1$ as $k \to \infty$ with continuous monitoring.

What guardrails cost

A rollout is rarely judged on one metric. There is the target metric and then a suite of guardrails — latency, error rate, cost, refusal rate — any of which can block the release.

PropositionThe sensitivity cost of a guardrail suite

Testing $m$ metrics and controlling the family-wise error rate at $\alpha$ by the Bonferroni correction means testing each at $\alpha/m$ . The required sample size per metric becomes

n_m \;=\; \frac{2\sigma^2\bigl(z_{1-\alpha/(2m)}+z_{1-\beta}\bigr)^2}{\Delta^2},

and since $z_{1-u} \sim \sqrt{2\ln(1/u)}$ as $u \to 0$ , $n_m$ grows like $\ln m$ .

Concretely, at $\alpha = 0.05$ and 80% power: one metric needs the constant $15.7$ ; ten metrics need $2(z_{0.9975}+0.84)^2 = 2(2.807+0.842)^2 \approx 26.6$ ; a hundred need $2(3.481+0.842)^2 \approx 37.4$ . Going from one guardrail to a hundred costs about $2.4\times$ the traffic — expensive, but logarithmic rather than linear, which is why large guardrail suites are affordable at all.

What this does not say

Nothing here addresses whether the metric measures what you care about. A canary that is statistically clean on latency, error rate and cost can still ship a model that is worse in ways none of those capture — and for generative systems the important regressions are usually of that kind. The statistics above bound the risk of being fooled by noise; they say nothing about the risk of measuring the wrong thing, which is the larger risk in practice and is not a statistical problem at all.

References

PaperR. Johari, P. Koomen, L. Pekelis, D. Walsh (2017). Peeking at A/B Tests: Why It Matters, and What to Do About ItQuantifies the error inflation from continuous monitoring and gives always-valid alternatives.
BookR. Kohavi, D. Tang, Y. Xu (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B TestingStandard reference for the practical design of the experiments modelled here.

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

PropositionSample size for a two-arm comparisonproof includedDetecting a difference of size Delta with 80% power at the 5% level needs about 16 sigma squared over Delta squared observations per arm.
CorollaryHow long a canary must runproof includedDuration is the required sample size divided by the traffic share allocated to the canary, so small canaries are slow detectors.
PropositionRepeated testing inflates the false-positive rateproof includedTesting at k interim points, each at level alpha, gives an overall false-positive probability that grows with k and tends to 1 under continuous monitoring.
PropositionThe sensitivity cost of a guardrail suiteproof includedControlling the family-wise error rate across m metrics by Bonferroni raises the required sample size by roughly a factor of log m over the single-metric case.

Problems (1)

All problems

How long must the canary run?NumericalIntermediate⏱ 30 min