Plug-in pass^3
0.512
INSTRUMENT · EVAL
reliability@k and pass^k Estimator
Estimate per-task pass^k and suite-level reliability@k from repeated eval runs, with Wilson and Student-t intervals and the unbiased pass@k counterpart.
pass^3 (this task)
0.512
Chance this task clears all 3 of 3 runs (plug-in). 95% interval [0.199, 0.777].
Unbiased pass^3
0.491
Supporting estimates at 95% confidence
| Metric | Estimate | Interval |
|---|---|---|
| Single-run pass rate (p̂) | 0.800 | [0.584, 0.919] |
| pass@3 (capability, unbiased) | 0.996 | n/a |
pass^k = p̂^k · reliability@k = mean over tasks of pass^kHow?
How this is calculated
reliability@k is the lane's name for the suite-level mean of per-task pass^k. It is our operating definition, not an industry-standard term. pass^k itself is the reliability counterpart of the established capability metric pass@k from Chen et al. (2021): where pass@k asks whether at least one of k runs succeeds, pass^k asks whether all k of them do.
Single-run pass rate. The point estimate is p̂ = c / n: successful runs over total runs. Its interval is the Wilson score interval, which behaves well near 0 and 1 and at small n where the normal approximation breaks down. Reporting Wilson over the textbook Wald interval follows the recommendation of Brown, Cai and DasGupta (2001).
pass^k, two ways. The plug-in estimate is p̂^k, and its interval is the Wilson bounds on p̂ carried through the map x to x^k, which is monotone so the bounds pass straight through. The plug-in is the estimate to report when you model k independent reruns, but a power of the point rate runs high on small samples. The unbiased estimate C(c, k) / C(n, k) is the chance that k runs drawn without replacement from the observed runs all pass. It needs k at most n and returns 0 when fewer than k runs passed. Report the unbiased value when you want a sample estimate with no upward bias; the two converge as n grows.
pass@k (capability). Alongside pass^k the tool reports the Chen et al. unbiased pass@k, 1 minus C(n - c, k) / C(n, k), so you can read capability and reliability from the same runs.
reliability@k (suite). Across a suite, reliability@k is the mean of each task's pass^k. Its interval is a Student-t interval across tasks with df = tasks minus 1, capturing how much pass^k varies from task to task. That interval leaves out the sampling noise inside each task, which more runs per task shrink; with fewer than two tasks there is no across-task interval to report. Every interval here uses your chosen confidence level, defaulting to 95% by the convention argued in Miller (2024).
Worked example (one task). With c = 16 successes in n = 20 runs, k = 3, at 95%:
- Single-run rate p̂ = 16 / 20 = 0.800, Wilson 95% interval [0.584, 0.919].
- pass^3 plug-in = 0.800^3 = 0.512. Cubing the full-precision Wilson bounds (shown rounded here as 0.584 and 0.919) gives the 95% interval [0.199, 0.777].
- pass^3 unbiased = (16·15·14) / (20·19·18) = 3360 / 6840 = 0.491.
- pass@3 capability = 1 minus (4·3·2) / (20·19·18) = 1 minus 24 / 6840 = 0.996.
The plug-in 0.512 sits above the unbiased 0.491 because a power of the point rate overstates on a 20-run sample. On this task the agent almost always solves it at least once in three tries (pass@3 = 0.996) yet clears all three only about half the time (pass^3 near 0.5).
Worked example (suite). Four tasks with counts 16/20, 18/20, 20/20 and 12/20 at k = 3 give per-task pass^3 of 0.512, 0.729, 1.000 and 0.216. Their mean is reliability@3 = 0.614. With t at df = 3 equal to 3.182 and a standard deviation of 0.332, the 95% interval works out to [0.086, 1.000], the upper bound clamped to 1. The wide interval is honest: four tasks that disagree this much cannot pin the suite mean tightly.
For the fuller argument on why a single pass rate understates what a suite of reruns tells you, see how to measure agent reliability past a single pass rate.
Formula: pass^k = p̂^k · reliability@k = mean over tasks of pass^k
Questions
What does reliability@k mean here?
reliability@k is the lane's operating name for the suite-level mean of per-task pass^k: run every task in a suite k times, compute each task's chance of passing all k runs, and average those. It is a definition we use for consistency across this work, not a standardized metric.
How is pass^k different from pass@k?
pass@k, from Chen et al. (2021), is a capability metric: the chance that at least one of k runs succeeds. pass^k is the reliability counterpart: the chance that all k runs succeed. A task can score high on pass@k and low on pass^k at the same time, which is exactly the gap reruns expose.
Which pass^k estimate should I report?
Report the plug-in p̂^k with its interval when you are modeling k independent reruns of the task. Report the unbiased C(c, k) / C(n, k) when you want a sample estimate with no upward bias and are not assuming independence beyond exchangeable runs. On small samples the plug-in runs high; the two converge as the number of runs grows.
Why does the suite interval sometimes reach 1.000?
The suite interval is a t-interval across tasks, then clamped to the [0, 1] range a probability lives in. With few tasks or tasks that disagree a lot, the raw interval can extend past 1 and gets clamped. A wide interval means the suite mean is not yet well pinned; add tasks or runs to tighten it.
How many runs per task do I need?
Enough that each task's pass rate has a usable interval, and enough tasks that the across-task interval is not dominated by a handful of outliers. The companion analysis on measuring agent reliability walks through the tradeoff.
Sources
- Chen et al. (2021), Evaluating Large Language Models Trained on Code (unbiased pass@k estimator)
- Miller (2024), Adding Error Bars to Evals
- Wilson (1927), Probable Inference, the Law of Succession, and Statistical Inference (the score interval)
- Brown, Cai and DasGupta (2001), Interval Estimation for a Binomial Proportion