TOOL | Eval statistics
No-regression verdict against a pre-set margin
Set a margin before the run and check paired pass/fail results against it: not worse, worse, or inconclusive, with one-sided bounds from a Newcombe paired interval and next-run sizing.
A test that found "no significant difference" can still hide a drop larger than your margin. Set a margin before the run, and this page checks whether paired pass/fail results rule out a drop larger than that amount, using one-sided bounds from a Newcombe paired interval. When the result is too close to call, the page sizes the next run. It reports the paired interval that our note on paired significance tests recommends.
Enter a margin above 0 and below 50 points before the data. The margin is the largest drop you are willing to accept.
Off by default. When on, the link also carries the counts on this page. Rows, labels and file names stay out of every link.
How this is calculated
The page builds a one-sided test from a two-sided Newcombe paired interval at twice the significance level. A 90% two-sided interval at alpha 0.05 gives each edge the force of a one-sided 95% bound.
Three verdicts against minus the margin: not worse if the lower bound sits above it, worse if the upper bound sits below it, and inconclusive if the interval contains it.
The interval is Newcombe's paired method 10 (Newcombe 1998). The mid-p McNemar line tests for any change in either direction and is context only (Fagerland, Lydersen and Laake 2013).
A significance test with enough power to detect a drop is weaker than checking whether the interval clears the margin (Schuirmann 1987). It can find no evidence of a drop while the interval still includes drops larger than your margin.
Cases for the next run use Wald planning for paired non-inferiority: add the normal quantiles for one minus alpha and for the power, square the sum, multiply by the discordance, divide by the square of the margin minus the absolute planned difference, and round up. Discordance is the share of cases that flipped between versions. When none flipped, the page asks for a planning estimate.
Questions
How should I choose the margin?
The margin is the largest drop in the pass rate you are willing to accept when you ship the new version. It comes from the business decision: how much worse is acceptable given what the new version gains (cost, speed, a new capability). Write it down and date it before the data arrives. A margin decided after seeing the results can be tuned to the outcome.
Why set the margin before the data?
Setting the margin after seeing the results lets you pick a number that makes the answer you want come out. That defeats the test. The export records when the margin was entered and when the data arrived. If the margin was set later, the export flags it.
Why is the interval 90% and not 95%?
The page builds a one-sided bound from each edge of a two-sided interval. A 90% two-sided interval at a one-sided 5% level gives each bound the same force as a standard 95% one-sided test. If you lower alpha to 0.025, the interval widens to 95% two-sided.
What should I do when the verdict is inconclusive?
Run the comparison again on more paired cases, with the margin you set unchanged. The page shows how many cases the next run needs at the power you chose. Lower power makes the count smaller and another inconclusive result more likely. Changing the margin now would be setting it after the data.
I ran a significance test and it found no difference. Why does this page say something different?
A significance test asks whether there is any detectable change. Finding no significant change means the data is consistent with no change, and it can also be consistent with a drop larger than your margin. This page asks a different question: is the entire interval above minus the margin? That question can have a clear answer even when the significance test is ambiguous.
Can I compare results from two different case sets?
This page requires the same cases on both versions. Pairing on the same cases removes differences in case difficulty from the comparison. Comparing different case sets needs an unpaired interval and a larger sample; that is a different method and is out of scope here.
How does this page differ from the release gate?
The release gate designer applies a pre-registered rule to hold or promote a release. This page checks a single margin against paired results and sizes the next run. To use these numbers in a gate, type them into the release gate designer.