reliability@k and pass^k

pass^k is the probability that an agent system succeeds on every one of k independent runs of a single task; its closed form is p^k under a stated per-run success rate p. reliability@k is the lane’s suite-level report of this consistency view: the mean of pass^k across a representative task suite. The per-task number falls as k grows, because demanding success on more runs is a harder bar. It is the consistency counterpart to pass@k, the capability metric from HumanEval-style code generation, where k samples are drawn and a task counts as solved if any one passes the unit tests (Chen et al., 2021, arXiv:2107.03374, preprint).

pass@k measures the best of k tries; reliability@k measures the tries you cannot afford to lose.

The gap is operational. Take a per-run success rate of 0.9 with k = 3:

• pass@3 reaches 1 - 0.1^3 = 0.999, since one passing try is enough.
• pass^3 is only 0.9^3 = 0.729, since all three runs have to hold.

Both figures are modeled from the stated rate for illustration only. The same system clears 0.999 when a single success counts, yet holds all three runs about seven times in ten.

At k = 1 the two metrics collapse: pass@1 and pass^1 both equal the single-run rate p, and the split only widens as you ask for more runs to hold.

Where pass@k comes from

pass@k answers a capability question. Sampling-based decoding draws many candidate programs per prompt, so a single-sample score understates what a model can produce when several attempts are allowed and a way exists to pick the winner. A pass@k figure climbs as k rises. reliability@k holds the same system to a stricter bar: whether all k runs succeed. A deployed agent that retries a flaky step, or runs one workflow for many users, sits on the pass^k side, where each added run is another chance to fail, and how many of those runs to plan for is what the repeated-run variance planner sizes.

Every pass^k needs a confidence interval

Both pass^k and reliability@k are estimated from a finite number of runs, so a bare point value claims more precision than the data hold. Report each with a confidence interval, and expect it to widen as the suite shrinks: ten runs pin pass^k far less tightly than two hundred. The interval is where a k-run number becomes decision-grade. The estimator below returns both the point value and its interval, and eval confidence interval covers how the band is built.

A single-run pass rate reports neither pass^k nor reliability@k. See how to measure agent reliability for the full k-run methodology and agent reliability testing for the protocol.