The agent reliability testing checklist your green eval skips

The eval passed. The dashboard is green. Then the agent ships, runs its twenty-step pipeline unattended, and completes cleanly only about a third of the time, because a single green run shows it can pass once while shipping depends on whether it passes step after step. At 0.95 per step, a twenty-step chain finishes end to end about 0.36 of the time (computed from stated assumptions, not measured), and that gap is the whole reason a green eval only opens a reliability decision the rest of this list has to close.

This page is the checks that turn the green run into a defensible ship call.

Before you ship, clear six checks: failure-mode coverage, a powered run count, significance-tested deltas, pass^k and reliability@k at your operating k, a cascade budget that survives the chain, and a judge calibrated against humans. Each check below carries a pass criterion and the calculator that computes it. Skip any one and the rest report tidy numbers that fall apart under repetition, so the whole list runs before the release, not after the incident.

What is a pre-flight agent reliability testing checklist?

A pre-flight agent reliability testing checklist is the fixed set of six checks a team runs against a release before an agent ships, each with its own pass criterion and calculator. The checks move the ship decision from single-run evidence to repeated-run evidence, spanning failure-mode coverage, statistical power, delta significance, pass^k and reliability@k, the end-to-end cascade budget, and judge calibration. Run in order, the list turns a green eval into a decision you can defend.

TABLEShow full table (6 rows)Showing full table (6 rows)

Check	What to verify	Pass criterion	Calculator
Failure-mode coverage	The suite exercises the failure modes your topology produces, mapped to a taxonomy	Every documented mode has at least one probing task; no mode ships untested	failure taxonomy, RAG chunk-size estimator
Trials and power	N sized from the smallest regression you would block on, at your target power	N clears the power calculation for your effect size and baseline rate	sample-size / power, pass-rate CI
Delta significance	The pass-rate change between two runs is tested on the discordant pairs	Paired McNemar p below your threshold, with a CI on the delta	McNemar test
pass^k and reliability@k	Per-task all-pass probability at your operating k, aggregated across the suite	reliability@k reported at the k your system runs before a human checks	reliability@k estimator
Cascade reliability budget	End-to-end success under compounded per-step reliability across the chain	Compounded success clears your production floor, with levers costed	cascade reliability
Judge calibration and bias	Judge-to-human agreement measured; position and verbosity bias tested	Cohen’s kappa in the substantial band; swap-consistency within tolerance	inter-rater, bias correction, position bias

Each section below takes one row, states the number you must walk in with, and the pass criterion that clears it. Where the interval math runs deep, the derivation lives in the pillar on how to test agent reliability beyond a single run and the section points to it; this page stays on what to record and which threshold to hold.

Cover the failure modes your architecture actually produces

A suite that only walks the happy path certifies the happy path. The failures that ship are the ones no task probed. So the first check is coverage: enumerate the failure modes your topology can produce, then confirm each one has a task that tries to trigger it.

The taxonomy is already built for you. MAST, the multi-agent system failure taxonomy, derived 14 recurring failure modes in 3 categories from 150 expert-annotated traces at inter-annotator kappa 0.88, then applied them across a 1,600-plus trace dataset spanning 7 frameworks (Cemri et al., Why Do Multi-Agent LLM Systems Fail?, arXiv:2503.13657, preprint, as of 2026-07).

Three families recur: system-design and specification faults, inter-agent misalignment, and task-verification gaps. Map each family onto your architecture. An orchestrator-worker system inherits the misalignment modes at every hand-off; a retrieval pipeline inherits verification gaps at the generation step, where a well-grounded prompt still produces an ungrounded answer.

For a retrieval system, coverage extends to the retrieval configuration itself, since a chunk boundary that splits the answer across two passages is a failure the generator cannot recover from; the RAG chunk-size estimator sizes that boundary, and how to evaluate a RAG pipeline covers the retrieval-versus-generation split. Instrument every trace well enough to pin each escape to the exact hop it slipped past, because a coverage report you cannot decompose tells you something broke without telling you where.

Pass: every failure mode your topology can produce has a probing task, or is logged as a known blind spot.

Size the run count before you trust a single number

A pass rate is one draw from a distribution, so a single run barely constrains the next. The check is that your N was chosen to resolve the decision you gate on. Sizing runs from a target regression uses the normal approximation to the binomial: to catch a drop of size d from a baseline p at 80% power and two-sided 5% significance, you need roughly (1.96 plus 0.84) squared times p times (1 minus p) over d squared runs per version (NIST/SEMATECH e-Handbook, Sample sizes required, primary, as of 2026-07).

Read the budget off the smallest regression you would actually block, at a 0.9 baseline and 80% power (computed from stated assumptions, not measured):

Those counts are two-sided, computed with the normal approximation and the variance taken at the baseline rate. Run one-sided, the shape a regression gate usually takes, and that same baseline-only approximation puts the five-point drop nearer 220; the canonical how many runs make a reliable eval computes about 250 for that case using NIST’s one-proportion form, a candidate scored against a fixed, trusted baseline, which carries the variance at both the baseline and the degraded rate and runs slightly higher, so where a released number has to hold, defer to the pillar.

Comparing two equally noisy runs instead, a fresh candidate against a fresh baseline, roughly doubles that per-arm budget, to about 540 runs on each side. That page derives both sides, the sample-size and power calculator returns N from your baseline and effect size, and the pass-rate CI calculator gives the interval on any rate you report.

Run thinner than the table and a real regression clears the gate more often than it trips it.

Pass: N meets or exceeds the count that powers detection of the smallest regression you would block, and every rate ships with its confidence interval.

Prove a delta is real before you gate on it

Two runs come back at 0.89 and 0.83. Did the new version really drop six points, or did the coin land differently? Eyeballing the gap treats sampling noise as signal. The check is a significance test on the pass-rate difference, run on the items that actually changed.

When both versions run against the same frozen inputs, the right test is paired, because most of the run-to-run noise is shared item difficulty that cancels in the difference. McNemar’s test looks only at the discordant pairs: items the old version passed and the new one failed (call it b) against items the new version passed and the old one failed (c). The statistic is (b minus c) squared over (b plus c).

Say across 200 items you see b = 18 regressions and c = 6 recoveries: the statistic is 12 squared over 24, which is 6.0, above the 3.84 threshold for one degree of freedom at the 5% level, so p is about 0.014 (schematic arithmetic on round inputs). The six-point drop is real. Had the split been 12 and 9, the statistic would be about 0.43, and the gap would be noise you must not gate a decision on.

Pass: every delta you gate on carries a paired McNemar p below your threshold and a CI on the difference, so a ranking survives only where that interval excludes zero. The McNemar test calculator takes your discordant counts and returns both, and is my eval statistically significant covers when to reach for the paired test versus a two-proportion comparison.

Report pass^k at the k your system actually runs

Two metrics share the name pass and pull in opposite directions. pass@k, from code generation, scores a task as solved when any one of k samples passes, so it climbs toward 1 as k grows and certifies best-of-k capability (Chen et al., Evaluating Large Language Models Trained on Code, arXiv:2107.03374, preprint, as of 2026-07). That is the right metric when a human picks the best of several drafts. An agent running unattended needs the consistency counterpart.

Call it pass^k: the probability that all k independent runs pass, which at a constant per-run rate p is p raised to the k. The all-pass rate falls fast as k grows, and that decline is what single-run reporting hides (computed from stated assumptions, not measured).

A 0.95 agent a human checks every run sits at 0.95; the same agent in the twenty-step chain from the opener sits at 0.36. The all-pass rate collapses as k climbs, which is the reliability a single run never reports.

Set k to the number of consecutive successes your product depends on before anything, a person, a verifier, or a hard gate, inspects the output. A reply a human reads every time sits at k of 1; an unattended nightly pipeline sits at k of its step count, where the matrix turns brutal.

Pass: report per-task pass^k and its suite mean as reliability@k at your operating k, with the independence assumption and task-to-task spread stated alongside. A healthy suite average can still bury a handful of hard tasks that rarely clear all k runs, so report the spread. The reliability@k estimator computes both with intervals, and how to measure agent reliability is the full derivation.

Budget reliability across the whole chain

Per-task numbers can all look fine while the assembled system fails, because reliability compounds down a chain. That 0.36 from the opener is the compounding at work: multiply 0.95 across twenty steps and the chain finishes about a third of the time, so steps that each look shippable assemble into a pipeline that misses two runs in three. The check is that your end-to-end reliability budget was computed across the actual length of your chain and clears your production floor.

Two forces sit inside that budget. The first is the compounding above: multiply the per-step success rates along the critical path, and the product is the ceiling on unattended completion. The second is propagation, the failure class that task eval never sees, where a mid-pipeline fault travels between agents. OWASP files this as a distinct agentic risk class, ASI08, describing false signals that cascade through automated pipelines with escalating impact (OWASP Top 10 for Agentic Applications, as of 2025-12); reading each OWASP agent item as a testable reliability control is the security-side companion to this budget.

The way to test it is fault injection: introduce a controlled fault at a known hop, trace its propagation radius, and report a containment rate with an interval built from enough repetitions to bound it.

Pass: compounded end-to-end reliability clears your product floor, and every multi-agent topology reports a fault-injection containment rate with an interval, per topology rather than blended. Where the budget falls short, the production levers (retry, fallback, checkpoint, human-gate, graceful degrade) are costed against the steps that most drag the product down, since a fault a sequential chain stops at one stage can spread across every worker a hub fans out to. The cascade reliability calculator does the per-step budget math, reliable AI agents in production lays out the lever table, and how a single fault propagates across topologies covers the injection mechanics.

Calibrate the judge before you trust its scores

Every number above assumes the thing scoring your runs is trustworthy. When an LLM grades the outputs, that assumption needs its own check, because an uncalibrated judge quietly corrupts the whole suite. The calibration is real work with a real payoff: a controlled study reported GPT-4 agreement with human preference near 85 percent, above the 81 percent agreement between two humans, which is what makes an LLM judge usable at all (Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023, arXiv:2306.05685, as of 2026-07).

The first sub-check is agreement. Score a small human-labeled calibration set with your judge and compute Cohen’s kappa against the human labels, which corrects raw agreement for chance. Landis and Koch’s benchmark places 0.61 to 0.80 in the substantial band and 0.81 to 1.00 in the almost-perfect band (The Measurement of Observer Agreement for Categorical Data, Biometrics 33:159-174, peer-reviewed, as of 2026-07). A judge below the substantial band is not calibrated for a release gate. The inter-rater reliability calculator computes kappa from your judge-versus-human confusion counts, and once you know the judge’s sensitivity and specificity, the bias-correction calculator recovers a bias-corrected accuracy with an interval instead of quoting the judge’s raw score.

The second sub-check is bias. The same study that established the judge’s usefulness also documented position, verbosity, and self-enhancement bias, where the judge favors the first option shown, the longer answer, or outputs resembling its own. Test for the one that distorts pairwise scoring most: swap the order of the two answers and re-judge. The position-bias calculator turns order-swap disagreement into a position-robustness rate, so a verdict that holds on both orders stays trustworthy.

Pass: judge-to-human Cohen’s kappa sits in the substantial band or higher, and a verdict holds when you swap the answer order, within your tolerance. The taxonomy of judge biases and the detection test for each lives in LLM-as-a-judge bias, the judge-reliability method in LLM as a judge, and how to report the corrected numbers in reporting LLM-as-a-judge evaluations.

Run the list in order, because each check feeds the next

The six checks are a sequence, and the order is load-bearing. Coverage decides what the suite even contains. Variance and power decide how many times you run it. Significance decides which deltas you are allowed to act on. pass^k reframes the rate into the quantity your product depends on. The cascade budget assembles the per-task numbers into an end-to-end claim. Judge calibration underwrites every score the earlier checks consumed. Run them in that order against a release:

1. Confirm coverage. Every failure mode your topology produces has a probing task, or is logged as a known blind spot.
2. Size N to the decision. The run count clears the power calculation for the smallest regression you would block on.
3. Test the deltas. Every gated pass-rate change carries a paired significance test and a CI on the difference.
4. Report pass^k at your k. Per-task all-pass probability and its suite mean, quoted at the k your system runs before a human checks.
5. Budget the chain. Compounded end-to-end reliability clears the floor, with propagation containment measured per topology.
6. Calibrate the judge. Kappa in the substantial band and swap-consistency within tolerance before any judge-scored number counts.

The instrument this site is building toward is a pre-launch reliability profiler, whose design intent is to run this checklist end to end and report pass^k and a containment rate with confidence intervals against field norms. Pre-launch means no such number is measured yet, so none is asserted here. Until it ships, the checklist runs by hand against the calculators above, and the pillar on testing agent reliability beyond a single run carries the statistics each check rests on. Carry one rule back to your release: a green eval is where this list starts, and a ship call is where it ends.