Skip to content
nonconvexcard

How long must the canary run?

NumericalIntermediate⏱ 30 minAI Systemscanarypowersample sizesequential testing

Problem

Total traffic is 2000 requests per second. A canary receives 5%. The metric of interest has mean mm and coefficient of variation 1.51.5.

  1. How long must the canary run to detect a 2% relative regression with 80% power at the 5% level?
  2. How long to detect a 0.5% regression?
  3. The team checks the dashboard every 10 minutes and ships when p<0.05p < 0.05. Estimate their actual false-positive rate over a 24-hour window, and say what to do instead.

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.