The design
INSTRUMENT | Eval statistics
Paired vs Independent: Choosing an Eval Comparison Design
5 cited sources
Six questions give you the design, the test that fits it, the rules that fired, and what the branch you did not pick costs in cases and model runs.
Before you run anything, decide whether every system will score the same cases. That one choice changes which test you can use and how many cases you need.
Once the runs exist and you are picking a test to fit data you already hold, the significance walkthrough is the page for that. This one runs earlier. It routes to the McNemar calculator and prices both designs against the sizing tool.
Showing your last valid result. Update the inputs above to recompute.
Rule that fired
What drove this
| Stage | What we asked | Your answer | Rule |
|---|
Every rule we checked, and where each one stopped
| Stage | Rule | What happened | Stopped at |
|---|
What each design costs
| Design | Cases | Model runs | Versus the other |
|---|
The counts below a pass/fail outcome do not carry over to this one. What pairing buys still holds: the variance of a paired estimate is the unpaired variance minus twice the covariance between the arms, divided by the number of cases, so the more the two systems agree case by case, the more pairing helps. Counts print only under a pass/fail outcome, because the two formulas here size proportions.
n_pairs = (z_a sqrt(psi) + z_b sqrt(psi - d^2))^2 / d^2How?
How this is calculated
Two passes, not one. The first pass reads whether the cases are shared and returns the design. The second reads that design, the number of systems and what a case produces, and returns the instrument. Running them together lets a two-system rule swallow a three-system answer, because the first matching rule wins before the arm count is ever read.
Pass one: the design
| Rule | When it fires | Design |
|---|---|---|
| D1 | the arms see different cases | independent |
| D2 | only some cases are scored under every system, and the cases outside the overlap are outside it by design | paired-on-overlap |
| D3 | only some cases are scored under every system, and the gap came from arm failures, or nobody knows where it came from | paired-on-overlap |
| D4 | every case is scored under every system | paired |
Pass two: the instrument, all sixteen cells
| Rule | Design | Systems | One case produces | The test |
|---|---|---|---|---|
| R1 | Paired | Two | Pass or fail | McNemar's test, exact binomial when fewer than 25 cases disagree |
| R2 | Paired | Two | A score | A paired bootstrap, or a paired t-test on the per-case difference |
| R3 | Paired | Two | A preference | A paired win rate with an interval around it |
| R4 | Paired | Two | A category | The Stuart-Maxwell test of marginal homogeneity (we ship no calculator for it) |
| R5 | Paired | Three or more | Pass or fail | Cochran's Q across all systems, then pairwise McNemar with a multiplicity correction |
| R6 | Paired | Three or more | A score | The Friedman test, then pairwise Wilcoxon signed-rank with a correction (we ship no calculator for it) |
| R7 | Paired | Three or more | A preference | An all-pairs win-rate matrix with one interval per pair, or a Bradley-Terry fit |
| R8 | Paired | Three or more | A category | Pairwise Stuart-Maxwell with a multiplicity correction (we ship no calculator for it) |
| R9 | Independent | Two | Pass or fail | A two-proportion z-test with a pooled null, or Fisher exact when either arm is small |
| R10 | Independent | Two | A score | A Welch two-sample t-test, or a two-sample bootstrap (we ship no calculator for it) |
| R11 | Independent | Two | A preference | No test. A preference judgment needs both outputs on one input (we ship no calculator for it) |
| R12 | Independent | Two | A category | A chi-square test of independence, or Fisher-Freeman-Halton when cells are sparse (we ship no calculator for it) |
| R13 | Independent | Three or more | Pass or fail | A chi-square test of homogeneity across the arms, then corrected pairwise two-proportion tests (we ship no calculator for it) |
| R14 | Independent | Three or more | A score | One-way ANOVA, or Kruskal-Wallis when the scores are ordinal, then corrected pairwise tests (we ship no calculator for it) |
| R15 | Independent | Three or more | A preference | No test. A preference judgment needs both outputs on one input (we ship no calculator for it) |
| R16 | Independent | Three or more | A category | A chi-square test on the arm-by-category table, then corrected pairwise tests (we ship no calculator for it) |
What an independent comparison costs. Two arms, each scored on its own cases, need (za sqrt(2 p̄(1-p̄)) + zb sqrt(p0(1-p0) + p1(1-p1)))2 / d2 cases per arm, where p1 is the baseline minus the difference you want to catch and p̄ is the average of the two. At a 90% baseline, a 5-point difference, 5% significance and 80% power that is 685.6, so 686 per arm: 1,372 cases and 1,372 model runs.
What a paired comparison costs. A paired design reads only the cases the systems disagree on, so its count turns on the disagreement rate rather than the pass rate: (za sqrt(psi) + zb sqrt(psi - d2))2 / d2 pairs. At 15% disagreement that is 468.6, so 469 cases. Each case runs under both systems, so 938 model runs: 66% fewer cases and 32% fewer runs than the independent design. At 30% disagreement it is 940 cases and 1,880 runs, which is 31% fewer cases but 37% MORE runs.
Where pairing stops helping. Those two crossovers are solved rather than quoted. Setting the paired count equal to the independent per-arm count gives the run-parity rate, about 21.9% disagreement at these inputs; setting it equal to the two-arm total gives the case-parity rate, about 44%. Both are recomputed from your own numbers. The anchor underneath them: if the two systems' per-case outcomes were independent of each other they would disagree on about 22.0% of cases at this baseline, which sits just above the run-parity rate. Independence is always the higher of the two, and at a wide difference it sits several points higher, so pairing saves runs only once the systems agree more than chance, which most systems built on the same data do.
Three or more systems. Both counts above are for one pair. A k-system comparison needs an omnibus test first and a multiplicity correction on every pairwise reading after it, and both raise the counts. The panel says so rather than stretching a two-arm number across k arms.
Rounding. Every count rounds up, because a fraction of a case cannot be run. The saved percentages are computed from the rounded counts so the table agrees with itself. Significance is fixed at 5% two-sided and power at 80%, matching the sizing tool this page links.
Formula: n_pairs = (z_a sqrt(psi) + z_b sqrt(psi - d^2))^2 / d^2
Questions
Why is pairing not free power?
Because it costs runs. A paired design runs every case under every system, so a case is two runs rather than one. It buys back more than that whenever the systems agree case by case, since the comparison then reads only the cases where they part company. Above roughly a fifth of cases disagreeing, the extra runs stop paying for themselves, and the page computes that crossover from your numbers rather than asserting a rule.
I only have published scores. What can I do?
You are in the independent branch whether you like it or not: two numbers produced on different work cannot be differenced case by case. Compare the rates directly, report an interval on each, and say plainly that the comparison is unpaired. If you can rerun even one system over the other's cases, that is worth more than any statistical repair.
One system errored on three cases. Are those cases still shared?
Score them as failures rather than dropping them. An error, a refusal and a timeout are all outcomes the system produced, and on a pass/fail measure they are failures. Dropping them looks tidy and quietly removes the cases that separated the systems, biasing the comparison toward whichever system failed loudly. If something genuinely cannot be scored, report attrition per arm and never publish the overlap-only result on its own.
Crossover trials worry about carryover. Should I?
Usually not. The reason a crossover trial fears carryover is that a subject remembers the first treatment. An offline eval run has no memory between cases as long as each run starts clean, so the same case can go to every system without one contaminating another. The exceptions are real though: a shared cache, a judge that sees both answers in one prompt, or a system that learns from the session.
What changes with three or more systems?
Two things. You run an omnibus test first, which asks whether any system differs at all, and only then compare pairs. And every pairwise reading after that needs a correction for how many pairs you looked at, or the chance of one false alarm somewhere in the set climbs well past the level you think you are testing at. The cost panel prices one pair and says so.
Two per-arm intervals do not overlap. Is that the answer?
No, and the reverse trap is worse: two intervals can overlap while the difference is still significant. A per-arm interval shows how precisely that one rate is known. The comparison is a separate test on the difference, and it has its own interval. Read the test, and use the per-arm intervals to say how well each rate is pinned down.
Is this about how the judge sees the work?
No. That is a different question, and the scoring-protocol chooser answers it: whether a judge grades one output at a time or two side by side. This page is about the experiment around that, which is whether every system sees the same cases. The two decisions are independent, and you need both.
What comes after this?
The design choice comes before the counts. Once it is settled, size the set with the golden-set size planner, and if each case runs more than once, work out how many times with the repeated-run planner. Settle the design before the runs exist: one picked afterward is a rationalization of the numbers you already have.
Sources
- Miller (2024), Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
- Bowyer, Aitchison and Ivanova (2025), Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints
- Resolution Diagnostics for Paired LLM Evaluation
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.3.3: comparing two proportions
- McNemar (1947), Note on the sampling error of the difference between correlated proportions or percentages