How long must the canary run?
NumericalIntermediate⏱ 30 minAI Systemscanarypowersample sizesequential testing
Problem
Total traffic is 2000 requests per second. A canary receives 5%. The metric of interest has mean and coefficient of variation .
- How long must the canary run to detect a 2% relative regression with 80% power at the 5% level?
- How long to detect a 0.5% regression?
- The team checks the dashboard every 10 minutes and ships when . Estimate their actual false-positive rate over a 24-hour window, and say what to do instead.
Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.
Solution
Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.