TOOL | Eval statistics
How long to run a canary release before you know it is worse
Enter a pass rate, the drop to catch and your canary traffic share. See how many hours the watch needs and a stopping rule you can check whenever you like without inflating false alarms.
You are rolling out a release to a slice of traffic and watching the pass rate. Built on our run-count research and the after-the-run significance test, this page returns the hours the canary needs and a stopping rule that stays valid however often you check. The sample-size calculator sizes a run to a fixed count; this page times a live rollout.
Showing your last valid result. Update the inputs above to recompute.
Hours to a decision
1.8
At 2,000 requests per hour with 5% on the canary, a 5-point drop takes about 1.8 hours and 185 canary requests to detect.
Planning time to decision
This is the point at which a canary whose true drop is exactly the target, tracking its expectation, crosses the stopping boundary. Roughly half of real canaries at that drop cross earlier and half later.
The chance column first shows at least 80% at hour 4.
Roll back
Feed the result to the release gate.
The plan
| Figure | Value | What it is |
|---|---|---|
| Canary requests needed | 185 | Requests the canary arm accumulates by the decision point. Rounded up independently, so the two arms can sum to one more than the total. |
| Control requests needed | 3,504 | Requests the control arm accumulates by the decision point. Shown only for concurrent designs. Rounded up independently from the canary arm. |
| Fixed-sample comparison | 3,140 | The run count a single-look test at 80% power would need for the same drop. Sized by the sample-size calculator. |
| Peeking price | 1.17x | The ratio of traffic the always-valid rule needs to the traffic the fixed-sample plan would need. A ratio of 1.20, for example, means checking continuously costs 20% more traffic than a single look. |
Watch table
The last column is a lower bound, because crossing at N implies crossing by N.
| Hour | Requests | Canary requests | Interval half-width (pp) | Chance of catching a 5-pp drop by then (at least) |
|---|---|---|---|---|
| 1 | 2,000 | 100 | 6.83 | 21% |
| 2 | 4,000 | 200 | 4.80 | 55% |
| 4 | 8,000 | 400 | 3.44 | 92% |
| 8 | 16,000 | 800 | 2.49 | >99% |
| 12 | 24,000 | 1,200 | 2.07 | >99% |
| 24 | 48,000 | 2,400 | 1.51 | >99% |
The drop that stops the watch at an hour equals the interval half-width at that hour.
Smallest detectable drop at the window
1.39 pp
The drop at which the stopping rule and the watch window are self-consistent. Drops smaller than this cannot be caught within the window.
Recommended baseline
| Candidate | Source | Observations | Age (days) | Standard-error inflation | Usable as a canary baseline |
|---|---|---|---|---|---|
| 1 |
Version the chosen baseline with the golden-set version tracker before the rollout starts.
Preregister the window, the target drop and the rule in the run register before traffic moves. Watch the canary. Write the rule this watch feeds in the release gate. Report the final rate with the pass-rate CI calculator.
These figures are a worked illustration at a 95% pass rate, a 5-point drop, 5% canary traffic and 2,000 requests per hour. Replace every input with your own numbers.
N_seq = min { N : h(N) <= d }, h(N) = sqrt( 2*sigma2*(N*tau^2 + sigma2) / (N^2 * tau^2) * log( sqrt(N*tau^2 + sigma2) / (alpha*sigma) ) )How?
How this is calculated
The variance. With a pass rate p0 and canary traffic share
w, the per-request variance of the rate difference between arms is
p0(1-p0) / (w(1-w)) under a concurrent control and
p0(1-p0) / w under a fixed baseline. The concurrent form
accounts for noise on both sides; the fixed form drops the control term
because the baseline is treated as exact. At a 95% pass rate and a 5% canary
share, the concurrent variance simplifies to exactly 1.0.
The stopping rule. The page uses a confidence sequence: an interval around the observed rate difference that covers the true difference at every sample size, simultaneously, with probability at least 1 - alpha. A fixed-sample test is valid at one pre-chosen sample size; a confidence sequence is valid at every sample size you pass through. You can check the canary numbers after every batch of traffic, and the false-alarm guarantee still holds. The rule has two conditions. Roll back when the entire interval sits below zero: the canary is worse, and the evidence clears the false-alarm budget. Clear the canary when the lower end of the interval sits above minus the drop you entered in the Smallest drop to catch field: no drop that large is consistent with the data. When both conditions hold at once, the clear wins, because a drop smaller than the one you asked to catch never triggers a rollback.
The guarantee. The probability that the interval ever fails to cover the true difference, across every point you check, is at most alpha. This is a finite-sample bound from Ville's inequality on the mixture likelihood ratio. It holds from the first request onward and does not depend on the sample growing large. Two papers carry the mechanism: Robbins (1970) proved the martingale inequality the bound rests on, and Howard, Ramdas, McAuliffe and Sekhon (2021) extended it to the time-uniform confidence sequences this page computes.
Hours to a decision. N_seq is the smallest total request count at which the half-width equals the detectable drop, found by bisection. Dividing by the request rate gives hours. The page also reports the fixed-sample count N_fixed (at 80% power, for the same drop) and their ratio: the peeking price. The ratio varies slowly with the inputs because its dominant term is a logarithm.
What the hours figure means. N_seq is the point at which a canary whose true drop is exactly delta, tracking its expectation, crosses the boundary. Roughly half of real canaries at that drop cross earlier and half later. At the defaults the first ladder hour where the chance column reaches 80% is hour 4.
The watch table. One row per checkpoint on a standard ladder (1, 2, 4, 8, 12 and 24 hours, truncated at the watch window, plus the window itself). Each row shows the interval half-width at that hour and a lower bound on the chance of catching the target drop by then. The bound is a lower bound because crossing at N implies crossing by N; below 0.5% the page shows "<1%" and above 99.5% it shows ">99%". Do not read a row's bound as the power of the sequential rule; the page has no exact sequential-power figure and does not claim one.
The smallest detectable drop. The page reports the smallest detectable drop at the window: the drop size at which the interval half-width equals the drop. Drops smaller than this cannot be caught within the window. It is found by bisection, because the half-width depends on the drop through the rule's own tuning parameter.
Baseline selection. The baseline-selection section
evaluates candidate baselines by usability and noise. A concurrent control,
a previous release measured in the same window, or rolling history
qualifies as a canary baseline when it shares the canary arm's population.
An offline golden set or a published figure does not qualify. For each
usable candidate, the section reports the standard-error inflation the
choice costs: sqrt(1 + canary observations / candidate
observations), as a percentage. At ten or more times the canary's
count, the inflation is under 5% and the baseline can be treated as fixed.
The recommended candidate is the usable one with the largest observation
count, ties broken by smallest age. Once chosen, the section writes the
baseline design and observation count into the main calculator. Version the
chosen baseline with the
golden-set version
tracker before the rollout starts.
Formula: N_seq = min { N : h(N) <= d }, h(N) = sqrt( 2*sigma2*(N*tau^2 + sigma2) / (N^2 * tau^2) * log( sqrt(N*tau^2 + sigma2) / (alpha*sigma) ) )
Questions
I sized my eval run up front. Why do I also need a canary rule?
The sample-size calculator sizes a run to a count decided before anything runs: you set the number, run that many cases and look at the result once. A release canary is different because results arrive continuously and you want to check the numbers as they come in. The stopping rule on this page stays valid at every sample size, so repeated checking costs no correction.
Does checking the numbers repeatedly inflate the false-alarm rate?
No. The stopping rule is a confidence sequence whose guarantee covers every sample size simultaneously. The probability that the interval ever excludes the true difference, across every look you take, is at most the alpha you set. The guarantee is a finite-sample bound, so it holds from the first request onward.
What does the peeking price mean?
It is the ratio of the traffic this always-valid rule needs to the traffic a single-look test at 80% statistical power would need for the same drop. At typical inputs the ratio is about 1.17, meaning continuous checking costs roughly 17% more traffic. The extra traffic pays for the width the confidence sequence carries to stay valid at all future sample sizes.
How do I choose the smallest drop to catch?
Enter the smallest decrease in the pass rate that would change your decision about the release. The watch window and the traffic cost both grow as the target drop shrinks, roughly as the square of 1/delta. If you are unsure, start with the effect size your acceptance criteria already name, or the smallest regression the release gate is set to catch.
When should I use a fixed baseline instead of a concurrent control?
A concurrent control (the default) uses the traffic outside the canary slice as a live comparison arm. It is the conservative choice because it accounts for noise on both sides. A fixed baseline treats a historical rate as exact, which is appropriate only when the baseline rests on many more observations than the canary will accumulate. The baseline-selection section reports the standard-error inflation each candidate costs; at ten or more times the canary's count, the inflation is under 5%.
The page says the window cannot catch my drop. What are my options?
Three levers move the boundary. Raising the canary traffic share reduces variance; a 50/50 split needs the least total traffic, and a smaller slice costs more. Accepting a larger detectable drop reduces the required traffic sharply; halving the drop roughly quadruples the cost. Extending the watch window gives the rule more time. The page recalculates as you change any of the three.