Pass-rate statistics calculators
Turn eval runs into defensible reliability numbers: confidence intervals on a pass rate, the runs a target precision needs, consistency across repeats, paired significance, and rank error bars.
What these numbers mean
You ran the eval, the number came back higher than last week, and someone asked whether the difference is real. A pass rate is one number read off a finite sample, so it moves when you run the suite again. Pass-rate statistics are the figures that say by how much: the width of the interval, the runs behind it, whether a task passes on every one of its k runs (pass^k, averaged across the suite as reliability@k), and whether two rates differ by more than noise.
| What you have | The instrument |
|---|---|
| Each task run several times, runs disagree | score repeated-run consistency |
| One pass rate and no error bar around it | put a confidence interval on it |
| An eval to plan and no run count yet | plan the design, size, and cost first |
| Two systems scored pass or fail on the same items | test the paired difference |
| Graded scores on the same items, two systems | compare paired graded scores |
| Two pass rates measured on separate case sets, not the same items | test the difference as independent samples |
| A leaderboard with two systems one place apart | put error bars on the ranking |
| A pass rate that reads high, on a test set you did not build | check the file for train test overlap |
| A measured rate you trust, and a decision riding on what it costs to be wrong | price the decision the rate feeds |
| A pass rate ready to gate a release, not just sit in a report | turn it into a ship-or-hold rule |
The methods here are the ones our own analyses use: which eval number needs which test maps the family end to end, how many runs a regression needs before you can catch it covers sizing a run, and testing whether an eval difference is real covers the paired comparisons.
The pairing that confuses people most is McNemar against the A/B comparator. Both want the same items scored twice, once per system. Use McNemar when every item is a pass or a fail and the only thing that moved is which items flipped. Use the A/B comparator when each item carries a graded score and you want the size of the average difference, with a bootstrap interval around it.
Tools in this topic
pass^k, pass@k and reliability@k Estimator: Chance That All k Runs Pass
Estimate per-task pass^k and suite-level reliability@k from repeated eval runs, with Wilson and Student-t confidence intervals and the unbiased pass@k capability counterpart.
Instrument | Pass-rate statisticsWilson and Clopper-Pearson Pass-Rate Confidence Interval Calculator
Turn an eval pass rate (k of n) into a defensible confidence interval: Wilson score, Clopper-Pearson exact, and Wald normal bounds side by side.
Instrument | Pass-rate statisticsEval Sample Size and Power Calculator
Find how many eval runs you need to detect a pass-rate drop at a target power, across two-arm, fixed-baseline, and paired McNemar designs.
Instrument | Pass-rate statisticsMcNemar Test Calculator for Paired Eval Runs
Run McNemar's paired significance test on two eval runs scored on the same items, with exact and chi-square p-values and a CI on the pass-rate difference.
Instrument | Pass-rate statisticsLLM Eval A/B Comparator: Paired Difference with Bootstrap CIs
Run an LLM eval A/B test on paired graded scores: mean score difference with a bootstrap confidence interval, paired effect size, and win/tie/loss evidence.
Instrument | Pass-rate statisticsBenchmark Rank Uncertainty Calculator: Score Intervals and Rank Ranges
Put error bars on an LLM benchmark leaderboard. Paste scores with item counts, correct-of-total, or standard errors and see which adjacent ranks are a statistical tie.
Instrument | Pass-rate statisticsIndependent Two-Proportion Test Calculator for Eval Arms
Compare two pass rates measured on different case sets: a pooled two-proportion z test, Fisher exact when an arm is small, and an interval on the difference.
Analyses that use these calculators
- Multi-agent failures
Limitations of multi-agent debate: what the paper found
What arXiv 2509.05396 found about the limitations of multi-agent debate: accuracy fell below a first-answer vote, models flipped correct answers to agree, and a correctness reward made it worse.
- RAG & retrieval
Is grep all you need? Agentic search vs vector retrieval
The PwC paper behind the grep debate: lexical search won every inline pairing on LongMemEval, lost five of ten once results moved to files, and the harness shifted accuracy as much as the retriever.
- Reliability testing
Why AI agents hang: timeouts, stalls, and stops that don't
Agent runs hang in three shapes: an unbounded wait, an inactivity timeout a slow stream keeps alive, and a cancellation the work declines. Verified framework defaults, and how to tell them apart.
- Eval statistics
Eval statistics: which number needs which test
The statistics an agent eval rests on, routed by the question in front of you: sizing before the run, the interval on a rate, a paired test on a delta, and agreement on the labels.
- Multi-agent failures
Multi-agent failure vocabulary: the eight terms, defined
Eight multi-agent failure terms, each keyed to the MAST failure mode underneath it, the boundary where the fault becomes visible, and the quantity you count once you instrument that boundary.
Sources
- Probable Inference, the Law of Succession, and Statistical Inference (Wilson score interval)
- Interval Estimation for a Binomial Proportion
- The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial (Clopper-Pearson interval)
- NIST/SEMATECH e-Handbook of Statistical Methods: Proportions
- Evaluating Large Language Models Trained on Code (pass@k)