LLM Eval A/B Comparator: Paired Difference with Bootstrap CIs

The paired design. Both columns must score the same items in the same order, because everything here runs on the per-item differences. Pairing subtracts each item's own difficulty out of the comparison, which is what makes a small eval set informative at all. If your outcomes are pass/fail rather than graded scores, the paired binary test belongs in the McNemar test calculator instead.

The bootstrap, and which interval is reported. The tool resamples the n differences with replacement B times (a case resample of the pairs) and reads confidence bounds off the resampled means. The headline interval is Efron's bias-corrected and accelerated (BCa) interval, which adjusts the plain percentile interval for median bias and for skew in the differences; the uncorrected percentile interval is shown one row below so the size of the correction is visible. Quantiles interpolate linearly between order statistics. Resampling uses a seeded deterministic generator, so the same data, seed, and resample count reproduce the same bounds exactly. In the rare case where the BCa correction is undefined, the tool reports the percentile interval and says so.

Effect size. Cohen's dz is the mean difference divided by the standard deviation of the differences: the paired-design effect size, on the standardized scale where 0.2, 0.5, and 0.8 are the conventional small, medium, and large landmarks. It is not comparable to a between-groups d, because the paired denominator excludes between-item spread. Its interval is the percentile interval of dz over the same resamples; BCa is reserved for the mean, where the jackknife acceleration is stable. With few pairs or coarse scores, some resamples can have zero spread, and the tool then withholds the dz interval rather than compute it from a censored set.

Win, tie, loss evidence. A pair counts as a candidate win when its difference exceeds the tie margin, a loss when it falls below the negative margin, and a tie in between. The sign test is the exact two-sided binomial test on wins versus losses with ties excluded, the standard treatment. The decisive win rate carries a Wilson score interval, and the rank-biserial correlation summarizes how one-sided the signed ranks of the differences are, from -1 to +1.

Ordinal scores. A mean of 1-to-5 judge ratings treats the step from 3 to 4 as equal to the step from 4 to 5, which the scale does not promise. The sign test, the win rate, and the rank-biserial correlation use only the direction and ordering of differences, so they survive that objection. When the mean-based and rank-based readings disagree on an ordinal scale, weight the rank-based ones.

Worked example. The preloaded 12 pairs give a mean difference of +0.358 with a 95% BCa interval of +0.142 to +0.55: the interval clears zero, so the candidate's advantage is unlikely to be item-sampling luck. The sign test on the same data reads 8 wins to 3 losses, p = 0.227, and fails to reach significance, because it discards the sizes of the differences and eleven decisive pairs are too few for direction alone to decide. Both readings are correct answers to different questions.

Honest limits. The interval covers one source of uncertainty: which items ended up in the eval set. It sees one score per item per system, so rerun-to-rerun noise inside either system is invisible to it; scoring each system once is a modeling choice this tool inherits from your data. Below roughly 20 pairs, bootstrap intervals tend to run narrow. Bounds also move slightly with the seed and resample count, which is ordinary Monte Carlo error; more resamples shrink it. The companion pieces on when an eval delta is statistically real and how many runs a reliable eval needs cover the decision this number feeds.

Formula: mean diff = mean of (candidate minus baseline); CI = BCa bootstrap; dz = mean/sd of differences

See Sources below