McNemar Test Calculator for Paired Eval Runs

McNemar's test compares two runs that scored the same items pass or fail. Lay the paired results in a 2x2 table: a items both runs passed, d items both failed, b items the candidate passed and the baseline failed, and c items the candidate failed and the baseline passed. The items both runs agreed on say nothing about which run is better, so the test reads only the two discordant cells, b and c.

Under the null hypothesis of no difference, each discordant item is a fair coin: it was equally likely to flip either way. So min(b, c) follows a binomial distribution Bin(b + c, 0.5), and the exact two-sided p-value is 2 × P(X ≤ min(b, c)). This calculator sums that probability in log space, so it stays accurate even when b + c is large.

The mid-p variant subtracts half the probability mass at min(b, c). It is less conservative than the exact test and sits closer to the nominal error rate on average.

The chi-square form is (b − c)² / (b + c) on one degree of freedom. Edwards' continuity correction replaces the numerator with max(0, |b − c| − 1)², which pulls the statistic toward the null and never reports evidence the raw gap does not support. The chi-square approximation is trustworthy once the discordant count b + c reaches about 25; below that, the exact binomial test is the one to trust, following the mlxtend evaluation guidance in the Sources below.

When you also enter the agreement counts a and d, the calculator reports the pass-rate difference delta = (b − c) / N, where N = a + b + c + d, with its confidence interval. The large-sample Wald interval is delta ± z × (1/N)·√(b + c − (b − c)² / N), which you can reproduce by hand. Its recommended companion is Newcombe's (1998) score interval for paired proportions, built from two Wilson intervals and the paired correlation. Newcombe's interval stays well behaved when the discordant count is small, the regime where the Wald interval undercovers.

Worked example. Two runs over 200 shared items. The candidate passed 174 of 200 (87.0%), the baseline passed 160 (80.0%), a raw gap of +7 points. 180 items agreed, so the test lives on the 20 that disagreed: b = 17 the candidate flipped to pass, c = 3 it flipped to fail, a net of 14.

• Chi-square: (17 − 3)² / 20 = 196 / 20 = 9.80, giving a two-sided p of 0.0017 on one degree of freedom.
• Continuity-corrected: (14 − 1)² / 20 = 169 / 20 = 8.45, p = 0.0037.
• Exact binomial: 2 × P(X ≤ 3) for X ~ Bin(20, 0.5) = 0.0026. With 20 discordant pairs under 25, this is the p-value to report.
• Delta: 14 / 200 = 0.070 = +7.0 points. Wald 95% interval: 0.070 ± 1.960 × (1/200)·√(20 − 14² / 200) = 0.070 ± 0.0427 = [+2.7, +11.3] points. Newcombe 95% interval: [+2.7, +11.6] points.

Both intervals clear zero and the exact p is below 0.05, so the +7-point gain is resolved at 95%. The full procedure, including the unpaired test that reverses this verdict on the identical data, is worked at is your eval difference statistically significant.

Formula: exact p = 2 · P(X ≤ min(b, c)), X ~ Bin(b + c, 0.5)

See Sources below