TOOL | Evaluation design
Write a CI eval gate with three verdicts and a false-block forecast
Set pass-rate floors and regression margins for PR, nightly and release tiers, pin a baseline by hash, quarantine flaky cases, and see how often the gate will block an unchanged version.
Set a pass-rate floor or a regression margin per CI tier, and the page writes a gate.json your CI job reads. Each check returns ship, block or inconclusive. Clopper-Pearson is the default: at small counts, Wilson can block more versions on the bar than its level allows. The page forecasts monthly false blocks. Our research on statistical significance in evals and reliable eval sizing is the evidence. The CI planner prices each tier; the release gate is the human sign-off.
Set a pass-rate floor or a regression margin on at least one tier to build the contract.
Off by default. When on, the link also carries the baseline counts and the dry-run counts. Names, paths, hashes, case ids and file contents stay out of every link.
Dry run
Type candidate counts for one tier and see the verdict the contract gives.
Commands
How this is calculated
The gate checks each tier against up to two rules: a pass-rate floor and a regression margin, and returns ship, block or inconclusive per check.
The case rule collapses repeats before counting. All: every repeat passes; most: more than half (a tie fails); any: at least one passes. A repeat with no verdict counts as a fail under count-as-fail and is dropped under leave-out; a case with no scored repeat is excluded and counted. At rate p over r repeats, the 'all' chance is p to the r, 'most' the binomial chance of more than half, 'any' 1 minus (1 minus p) to the r. Repeats are assumed independent; shared state makes repeats pass or fail together, shifting the 'all' and 'any' chances.
The floor check builds a one-sided interval around the candidate's pass rate (each end is a one-sided bound at the level). It reads ship when the interval's lower end clears the bar, block when the upper end sits below it, and inconclusive when the interval straddles the bar. Clopper-Pearson inverts the binomial distribution and keeps the false-block rate at or under one minus the confidence level (5% at 95%) at every case count (Clopper and Pearson, 1934). Wilson is narrower, and at small counts its rate can run above that. At 40 cases, a 70% floor and a 95% level (the returns-desk example), Wilson's exact false-block rate is 6.33%, against Clopper-Pearson's 3.20%. At 400 cases with the same floor, both converge near 4.66%.
The regression check pairs each case by its id across the candidate and baseline runs under the same grader, case rule and repeats. It builds a two-sided Newcombe method 10 interval for the paired difference (candidate minus baseline), at the two-sided level that matches the one-sided level (90% for 95%). It reads ship when the interval's lower end stays above a drop of the margin, block when its upper end sits below that drop, and inconclusive otherwise (Newcombe, 1998).
The false-block probability at the bar is the exact binomial probability that a version whose true rate sits on the floor passes on no more cases than the largest blocking count.
The per-run forecast assumes the baseline pass rate holds. For the floor: exact binomial probabilities of ship, block and inconclusive at that rate. For the regression: a normal approximation with standard error equal to the square root of the rerun disagreement over the case count.
The nominal false-block ceiling is the number of runs per month times the significance level times the number of checks in the tier. Beside it sits the floor check's exact false blocks a month for the chosen method. Per check, Wilson's exact rate can run above the level at small counts, so on a tier with a floor alone its exact figure can pass the nominal one.
Your CI script applies the verdict rule and the exit codes the contract states. The dry run shows the verdict the script should return for given counts.
Questions
Why does the gate return three verdicts?
A small run can land between passing and failing. The interval around the pass rate straddles the bar, so the true rate could sit on either side. A regression check without a baseline has nothing to compare against. Both cases read inconclusive, and the contract says what happens: fail the job, or pass with a warning. The release tier defaults to fail, because shipping on a run that could not settle the bar is a risk the team should see.
What should each tier do with an inconclusive result?
A PR gate runs often and blocks a merge, so passing with a warning keeps the team moving. A nightly gate reports, so passing with a warning logs the result without failing a scheduled job. A release gate is the last check before shipping; failing on inconclusive forces a rerun or a larger case set before the release goes out.
Why pin the baseline by hash?
The gate reads the baseline file from a path the CI job knows. If someone replaces the file with a different run, the pass counts change and the regression check compares against the wrong numbers. The SHA-256 hash catches any change, including reordered rows, and the CI job refuses to run when the hash does not match.
Why use a regression margin?
A significance test asks whether the drop is nonzero. A margin asks whether the drop exceeds a threshold the team set in advance. A large enough run makes any nonzero drop significant, and a small run can miss a meaningful drop. The margin puts the team's tolerance into the contract.
Why can Wilson block more often than the level allows?
Wilson's interval is built from an approximation to the binomial distribution, and at small case counts that approximation is loose. At 40 cases, a 70% floor and a 95% level, a version sitting exactly on the floor is blocked 6.33% of the time, above the 5% the level allows. Clopper-Pearson inverts the exact binomial, so its false-block rate stays at or under 5% at every count.
How do flaky cases interact with the case rule?
One flip is a reason to measure the case's rate. Measure the pass share and interval first (Run rows shows each case's). Five repeats cannot tell a 4-of-5 case, whose rate could run from 37.6% to 96.4%, from one at 90%; the line under each tier shows what the rule lets through. Quarantine after measuring; a quarantined case keeps running as watched so a drop still shows. The Reruns of one failing case rule on the planner decides whether it dropped. Measuring first and keeping a quarantined case as watched is our synthesis.
How does this page differ from the CI planner and the release gate?
The CI planner prices each tier in minutes and money and shows what each tier can detect. The release gate records one human decision with its sign-off. This page writes the rule a CI job applies on every run and forecasts how often that rule blocks an unchanged version.
Why does this page produce no CI YAML?
Every CI provider has its own configuration format, authentication model and caching strategy. A generated YAML file would need updating with every provider change. The page writes the contract the CI job reads and gives two commands the job runs. The wiring between them is specific to your provider and pipeline.