Wilson and Clopper-Pearson Pass-Rate Confidence Interval Calculator

This tool reports a Wilson score interval by default: the range that brackets the true pass rate p behind an observed c passes in n independent runs, at the confidence level you pick. It also shows the Clopper-Pearson exact interval and the Wald normal approximation on the same inputs, so you can see where they agree and where they part.

Every method assumes each run is an independent Bernoulli trial at one fixed pass probability. That is the binomial model behind an eval confidence interval. The point estimate c/n is one draw from that distribution; the interval is the width around it that the evidence supports.

Wilson score interval (the default). With z the two-sided normal quantile for your confidence level, the interval centers and margins are:

center = (p̂ + z²/2n) / (1 + z²/n)

margin = (z / (1 + z²/n)) · √(p̂(1−p̂)/n + z²/4n²)

Worked example, c = 30, n = 100, 95% (z ≈ 1.9600, p̂ = 0.30, z² = 3.8415): center = (0.30 + 3.8415/200) / (1 + 3.8415/100) = 0.31921 / 1.03841 = 0.3074; margin = (1.9600 / 1.03841) · √(0.30·0.70/100 + 3.8415/40000) = 1.8875 · √0.0021960 = 1.8875 · 0.046862 = 0.0884. The interval is 0.3074 ± 0.0884, or 21.9% to 39.6%. Wilson stays inside 0% and 100% and returns a real width even at a 0% or 100% observed rate, which is why it is the default here.

Clopper-Pearson exact interval. The bounds are Beta quantiles, so the interval has at least the nominal coverage (it errs wide rather than narrow):

lower = BetaInv(α/2; c, n−c+1), upper = BetaInv(1−α/2; c+1, n−c), α = 1 − confidence

For the same c = 30, n = 100, 95% case: lower = BetaInv(0.025; 30, 71) = 0.2124, upper = BetaInv(0.975; 31, 70) = 0.3998, or 21.2% to 40.0%. Report this when you need a guaranteed-coverage bound and can accept a wider interval.

Wald normal approximation. The textbook bound is p̂ ± z·√(p̂(1−p̂)/n), which for c = 30, n = 100, 95% gives 0.30 ± 0.0898, or 21.0% to 39.0%. It is shown for contrast and carries a guard. The Wald interval collapses to a single point at a 0% or 100% observed rate, and it can fall outside 0% to 100%. This tool flags it as unreliable when either the observed passes or the observed failures fall below 10, the success-failure count condition. That floor of 10 is a common convention; Brown, Cai and DasGupta (2001) show Wald coverage stays erratic even at large n for rates near 0 or 1, so treat the floor as a minimum rather than a guarantee.

What the confidence level means, and what it assumes. A 95% two-sided interval is a frequentist statement about the procedure: across many repeated suites, about 95% of the intervals it builds would contain the true rate. It is not a 95% probability that this one interval holds the true rate. The binomial model also assumes runs are independent and identically distributed. Correlated seeds, a shared prompt, or a heterogeneous task mix break that assumption, and the real uncertainty is then wider than any bound here. Add the run-to-run spread from independent seeds on top of the task-set width this tool computes.

Runs that never got a score. A harness that drops a timed-out, refused or unparseable run before computing the rate answers a narrower question than the report implies. The unscored-runs panel prints answer coverage beside every rate, both counts showing, so the reader can see how much of the sample the rate rests on. With U the runs that came back without a score, S = c + f the answers returned and N = S + U the requests sent, the same counts support three readings:

scored only = c / S

all attempted = c / N (U counted as failures)

bound pair = [c / N, (c + U) / N] (the upper edge counts U as passes)

Each reading carries the same Wilson and Clopper-Pearson bounds as the default panel, and the envelope is the Wilson pair: the lower bound of the all-attempted reading to the upper bound of the upper edge. At S = 0 the scored-only rate is suppressed rather than published as 0%, because an empty denominator carries no rate, which is the cost of conditioning on the answered subset. Runs the model declined are one of the four classes counted here, beside timeouts, early stops and unparseable output. Counting unparseable runs here tells you how much of the sample they consumed. Measuring and pricing the structured-output failure rate is a different job.

How many clean runs a floor needs. The runs-for-a-claim panel solves the same exact bound for the run count. With C the ONE-SIDED confidence and alpha = 1 - C, the lower bound is:

R_L(0, n, C) = 0; R_L(k, n, C) = BetaInv(alpha; k, n-k+1); R_L(n, n, C) = alpha^(1/n)

zero-failure count: n = ceil( ln(1 - C) / ln(R) )

With failures budgeted there is no closed form, so the count is searched by doubling and bisection on the same bound, which rises with n at a fixed failure count. A 95% one-sided floor is the lower edge of a 90% two-sided interval, so the two directions on this page are not interchangeable: 299 clean runs demonstrate a 99% floor at 95% one-sided confidence, where reading the same level two-sided asks for 367. That is why the claim panel labels its confidence one-sided, and why it prints the two-sided Clopper-Pearson lower bound on the same data beside its own.

Formula: Wilson: (p̂ + z²/2n ± z·√(p̂(1−p̂)/n + z²/4n²)) / (1 + z²/n)

See Sources below