Independent Two-Proportion Test Calculator for Eval Arms

Two arms, each entered as cases passed out of cases run. Under the null hypothesis that both arms have the same true rate, the best estimate of that shared rate is the pooled proportion p̄ = (x₁ + x₂) / (n₁ + n₂), and the standard error of the difference is √(p̄(1 − p̄)(1/n₁ + 1/n₂)). Pooling is what makes this a test rather than a description: an unpooled standard error estimates each arm separately and answers a different question. On the default example the two differ in the second decimal place, 1.98 against 2.00, and both clear 0.05, so a wrong denominator here is invisible in the verdict.

The two-sided p-value is erfc(|z| / √2), evaluated directly. The algebraically identical 2 × (1 − Φ(|z|)) is a different number in floating point: at z = 10 the subtraction returns exactly 0, so a real result would print as an impossible one.

Which test runs. The normal approximation needs enough expected cases in every cell. This page prints all four expected counts under the pooled null, n₁p̄, n₁(1 − p̄), n₂p̄ and n₂(1 − p̄), and applies Cochran's rule: run the pooled z when the smallest is 5 or more, and Fisher's exact test otherwise. Fisher's exact conditions on both margins and sums the hypergeometric probability of every table at least as extreme as the one you have, in the two-sided sense of every table whose probability does not exceed the observed table's. The rule is on the page so you can see which side of it your counts fall on.

The interval on the difference. The recommended bound is Newcombe's square-and-add construction on the two Wilson intervals: with (l₁, u₁) and (l₂, u₂) the per-arm bounds and θ = p₁ − p₂, the interval runs from θ − √((p₁ − l₁)² + (u₂ − p₂)²) to θ + √((u₁ − p₁)² + (p₂ − l₂)²). It holds up at small counts and at rates near 0 or 1, where the textbook Wald interval does not. The Wald interval, θ ± z√(p₁(1 − p₁)/n₁ + p₂(1 − p₂)/n₂) clipped to the possible range, is shown because you can reproduce it by hand, never as the reported bound. The field has not settled this question; Miettinen and Nurminen's score interval is the live alternative, and moving to it would want the paired page's ranking revisited at the same time.

Reading several comparisons. Setting the comparison count above 1 divides the threshold by that count, the Bonferroni correction. It moves the bar and nothing else: the p-value is unchanged, the test is unchanged, and the interval keeps its full stated confidence level.

Worked example A, the default: 90 of 100 against 80 of 100 at 95%.

• p̄ = (90 + 80) / (100 + 100) = 0.85, so the expected counts are 85.0, 15.0, 85.0 and 15.0. The smallest is 15.0, well above 5, so the pooled z runs.
• Standard error = √(0.85 × 0.15 × 0.02) = 0.05049752, so z = 0.10 / 0.05049752 = 1.98, and the two-sided p is 0.0477. That clears 0.05, so the gap is resolved.
• Newcombe 95% interval on the difference: [+0.01, +19.9] pts. Wald for contrast: [+0.2, +19.8] pts. Fisher's exact, run as a cross-check, gives 0.0734.
• Per-arm intervals: [82.6, 94.5]% and [71.1, 86.7]%. They overlap between 82.6% and 86.7%, and the difference is resolved anyway. That is the whole argument for running a test instead of eyeballing two bars.

Worked example B, the small-arm switch: 9 of 10 against 4 of 10 at 95%.

• p̄ = 0.65, so the expected counts are 6.5, 3.5, 6.5 and 3.5. The smallest is 3.5, under 5, so Fisher's exact test is the reading.
• Fisher's exact gives 0.0573, which does not clear 0.05. The pooled z on the same counts would have been 2.34 with p = 0.0191, resolved. Printing all three figures is why the selection rule is visible: the reading is the test the rule picks, never the smaller p.
• The Newcombe interval is [+8.2, +74.6] pts, which excludes zero while the test reads unresolved. Both are on the page. An exact test is conservative by construction, and a score-based interval is not its dual, so at counts this small the pair can land either side of the line. The honest reading is that the evidence is thin: 10 cases per arm is not enough to settle a 50-point gap.

Formula: z = (p1 − p2) / √(p̄(1 − p̄)(1/n1 + 1/n2))

See Sources below