Wilson and Clopper-Pearson Pass-Rate Confidence Interval Calculator

This tool reports a Wilson score interval by default: the range that brackets the true pass rate p behind an observed c passes in n independent runs, at the confidence level you pick. It also shows the Clopper-Pearson exact interval and the Wald normal approximation on the same inputs, so you can see where they agree and where they part.

Every method assumes each run is an independent Bernoulli trial at one fixed pass probability. That is the binomial model behind an eval confidence interval. The point estimate c/n is one draw from that distribution; the interval is the width around it that the evidence supports.

Wilson score interval (the default). With z the two-sided normal quantile for your confidence level, the interval centers and margins are:

center = (p̂ + z²/2n) / (1 + z²/n)

margin = (z / (1 + z²/n)) · √(p̂(1−p̂)/n + z²/4n²)

Worked example, c = 30, n = 100, 95% (z ≈ 1.9600, p̂ = 0.30, z² = 3.8415): center = (0.30 + 3.8415/200) / (1 + 3.8415/100) = 0.31921 / 1.03841 = 0.3074; margin = (1.9600 / 1.03841) · √(0.30·0.70/100 + 3.8415/40000) = 1.8875 · √0.0021960 = 1.8875 · 0.046862 = 0.0884. The interval is 0.3074 ± 0.0884, or 21.9% to 39.6%. Wilson stays inside 0% and 100% and returns a real width even at a 0% or 100% observed rate, which is why it is the default here.

Clopper-Pearson exact interval. The bounds are Beta quantiles, so the interval has at least the nominal coverage (it errs wide rather than narrow):

lower = BetaInv(α/2; c, n−c+1), upper = BetaInv(1−α/2; c+1, n−c), α = 1 − confidence

For the same c = 30, n = 100, 95% case: lower = BetaInv(0.025; 30, 71) = 0.2124, upper = BetaInv(0.975; 31, 70) = 0.3998, or 21.2% to 40.0%. Report this when you need a guaranteed-coverage bound and can accept a wider interval.

Wald normal approximation. The textbook bound is p̂ ± z·√(p̂(1−p̂)/n), which for c = 30, n = 100, 95% gives 0.30 ± 0.0898, or 21.0% to 39.0%. It is shown for contrast and carries a guard. The Wald interval collapses to a single point at a 0% or 100% observed rate, and it can fall outside 0% to 100%. This tool flags it as unreliable when either the observed passes or the observed failures fall below 10, the success-failure count condition. That floor of 10 is a common convention; Brown, Cai and DasGupta (2001) show Wald coverage stays erratic even at large n for rates near 0 or 1, so treat the floor as a minimum rather than a guarantee.

The Jeffreys interval uses a Beta(c + 1/2, n - c + 1/2) posterior with the Jeffreys prior. Its bounds are the alpha/2 and 1 - alpha/2 quantiles of that distribution. At c = 0 the lower bound is set to 0 and at c = n the upper bound is set to 1, following the modification in Brown, Cai and DasGupta (2001). Both Wilson and Jeffreys are reasonable choices for an estimate, and at moderate sample sizes they return similar intervals. On the example's 12 failures in 400, the Jeffreys failure rate interval is 1.65% to 5.02%.

For an estimate, Wilson is the default, and Jeffreys is printed beside it at every sample size as an equally defensible alternative. For a bound claim, the page recommends Clopper-Pearson one-sided, whose coverage holds at every true rate, and names Wilson one-sided as the narrower alternative. A bound claim routes to the Runs for a claim panel, which prints the one-sided readout.

Coverage at a given true rate p is the probability that the method's interval contains p. For each method and for p on a grid from 0.005 to 0.995 in steps of 0.005, the page sums 1[L(c) <= p <= U(c)] Bin(c; n, p) over c = 0 to n. Coverage above 5,000 runs is not drawn because all four methods converge on that grid. At 100 runs and 95%, the lowest coverage on the grid is 91.0% for Wilson, 95.4% for Clopper-Pearson, 88.1% for Jeffreys and 39.4% for Wald.

Grouped or clustered runs inflate the variance of the point estimate (Miller, 2024). With m runs per group and within-group correlation rho, the design effect is 1 + (m - 1) rho, and the effective number of runs is n divided by that factor. The page computes the interval at the effective count. On the example with groups of 5 and rho 0.15, the design effect is 1.6 and the effective count is 250 runs. When the effective-n adjustment reverses a bound claim that held under independence, the page says so and shows both floors.

The relation between the two panels: a two-sided interval answers "where does the rate plausibly sit?" and a one-sided bound answers "can I defend a floor of at least 95%?" The two-sided upper edge sits farther from the observed rate than the one-sided bound at the same nominal level, so a bar that sits inside the two-sided interval can still clear the one-sided bound. That is why the page routes a bound claim to the Runs for a claim panel.

What the confidence level means, and what it assumes. A 95% two-sided interval is a frequentist statement about the procedure: across many repeated suites, about 95% of the intervals it builds would contain the true rate. It is not a 95% probability that this one interval holds the true rate. The binomial model also assumes runs are independent and identically distributed. Correlated seeds, a shared prompt, or a heterogeneous task mix break that assumption, and the real uncertainty is then wider than any bound here. Add the run-to-run spread from independent seeds on top of the task-set width this tool computes.

Runs that never got a score. A harness that drops a timed-out, refused or unparseable run before computing the rate answers a narrower question than the report implies. The unscored-runs panel prints answer coverage beside every rate, both counts showing, so the reader can see how much of the sample the rate rests on. With U the runs that came back without a score, S = c + f the answers returned and N = S + U the requests sent, the same counts support three readings:

scored only = c / S

all attempted = c / N (U counted as failures)

bound pair = [c / N, (c + U) / N] (the upper edge counts U as passes)

Each reading carries the same Wilson and Clopper-Pearson bounds as the default panel, and the envelope is the Wilson pair: the lower bound of the all-attempted reading to the upper bound of the upper edge. At S = 0 the scored-only rate is suppressed rather than published as 0%, because an empty denominator carries no rate, which is the cost of conditioning on the answered subset. Runs the model declined are one of the four classes counted here, beside timeouts, early stops and unparseable output. Counting unparseable runs here tells you how much of the sample they consumed. Measuring and pricing the structured-output failure rate is a different job.

How many clean runs a floor needs. The runs-for-a-claim panel solves the same exact bound for the run count. With C the ONE-SIDED confidence and alpha = 1 - C, the lower bound is:

R_L(0, n, C) = 0; R_L(k, n, C) = BetaInv(alpha; k, n-k+1); R_L(n, n, C) = alpha^(1/n)

zero-failure count: n = ceil( ln(1 - C) / ln(R) )

With failures budgeted there is no closed form, so the count is searched by doubling and bisection on the same bound, which rises with n at a fixed failure count. A 95% one-sided floor is the lower edge of a 90% two-sided interval, so the two directions on this page are not interchangeable: 299 clean runs demonstrate a 99% floor at 95% one-sided confidence, where reading the same level two-sided asks for 367. That is why the claim panel labels its confidence one-sided, and why it prints the two-sided Clopper-Pearson lower bound on the same data beside its own.

What a clean result proves. The highest failure rate n clean runs still allow is one minus the one-sided Clopper-Pearson floor: 1 minus the nth root of the significance level. The chance n runs at rate q show at least one failure is 1 minus (1 minus q) to the n. Clean runs for a C chance to see one: ceiling of ln(1 minus C) divided by ln(1 minus q). On the 200-of-200 example, 200 clean runs at 95% one-sided still allow up to 1.4867%. At 1% they show a failure 86.6% of the time. A 95% chance takes 299 runs. The rule of three approximates it as 3 divided by n: 1.5% at 200, against 1.4867%.

Both risks of a pass rule. A rule "at most c failures in n runs" passes a system at true failure rate q with the sum, j from 0 to c, of the binomial probability at j, computed through the regularized incomplete beta. The point-estimate rule takes c as n minus the ceiling of R times n. The one-sided floor rule takes the largest c whose one-sided Clopper-Pearson floor still reaches R, so it passes a system at exactly R with chance at most one minus the confidence level. The table shows exact binomial chances at true failure rates of 1.5, 1.25, 1, 0.75, 0.5 and 0.25 times the target's complement. On the 589-of-601 example at 98%: the point-estimate rule (at least 589 of 601) passes a true-98% system 57.4% of the time, the one-sided floor rule (at most 6 failures) 4.4%. In acceptance sampling these are called operating characteristics.

A plan checks n outputs and accepts the batch if at most c fail. The chance of acceptance at a true failure rate p is the sum, from j equals 0 to c, of the binomial probability at j. This is the method of NIST/SEMATECH e-Handbook section 6.2, applied to a batch of model outputs.

The smallest plan is the one with the fewest checks whose two risks both stay at or under the maxima you set. The page prints a line whenever a risk lands above its maximum. Under a random draw, where the plan meets its condition (allowed failures at or above the expected count at the good rate, at or below it at the bad rate), the binomial risk is at or above the exact figure for this batch. A zero-acceptance plan prints the producer's risk as the exact figure.

A random draw across the batch gives the binomial formula above: prompt families do not matter because each output is drawn independently of its family. Under a family draw, the number of failures from each drawn family follows a beta-binomial distribution with the similarity entered (m outputs checked per family), and the total is the sum over the families drawn. Applying the lot method to a batch of model outputs and pairing it with a family-draw model is our synthesis. The figures hold under that model at the similarity entered.

On the returns-desk example, a random draw across the batch needs 65 checks with at most 2 failures. Checking ten outputs per template at similarity 0.2 needs 171 checks with at most 6 failures. The same 65-check plan, if checked by template, passes an 8% batch about a quarter of the time. With similarity unknown, the plan is sized as if each drawn family fails or passes whole: 641 checks with at most 20 failures, and the printed risks are their largest values over the similarity.

ISO 2859-1 and ANSI/ASQ Z1.4 index plans by the good-batch failure rate (AQL) for a stream of lots with switching rules. This mode sizes one batch from two stated rates.

The verdict is the rule the plan set before checking. The two risks hold only for a plan fixed before the count was made. The early reject holds when the checking order was fixed before checking began: once more than c outputs have failed, the outputs left cannot change the outcome.

Formula: Wilson: (p̂ + z²/2n ± z·√(p̂(1−p̂)/n + z²/4n²)) / (1 + z²/n)

See Sources below