INSTRUMENT · EVAL
System Reliability Calculator for Multi-Step Agents
Compute the end-to-end success rate of a multi-step agent chain from its per-step reliability, and the per-step budget your target reliability demands.
35.8%
0.95^20 = 0.358 end to end over 20 steps
What one lever, applied to every step, buys.
| Approach | Effective per-step | End-to-end |
|---|
Per-step reliabilities and the weakest link.
| Step | Per-step | Note |
|---|
The per-step reliability budget for your target.
| Metric | Value |
|---|
end-to-end = p^N · required per-step = target^(1/N) · list = product of per-step ratesHow?
How this is calculated
The model treats a workflow as steps running in series: the end-to-end result lands only when every step holds. Give each step an independent success probability p, and a chain of N steps succeeds end to end with probability p raised to the N. That is the sequential-chain worst case, and it is the assumption baked into every number here.
Worked example. At 0.95 per step over 20 steps, 0.95^20 = 0.358, so a chain of twenty 95%-reliable steps succeeds end to end about 35.8% of the time. Push each step to 0.99 and 0.99^20 = 0.818; let it slip to 0.90 and 0.90^20 = 0.122.
Solving for the budget. Invert the curve and it becomes a budget. To hold a twenty-step chain to a 90% end-to-end target, every step must clear 0.90^(1/20) = 0.9947; a 99% target demands 0.99^(1/20) = 0.9995 per step. That is the per-step reliability budget the target forces.
A heterogeneous list. The per-step list multiplies a mixed chain: end-to-end is the product of the entered rates, and the lowest rate is flagged as the weakest link, where one lever buys the most.
What a lever buys. Uniform mode also shows what one lever, applied to every step, buys. One retry raises a step to 1-(1-p)^2 = 0.9975; two retries to 1-(1-p)^3 = 0.999875; a fallback that recovers half of a step's failures to 1-(1-p)(1-0.5) = 0.975. Each effective per-step rate is raised to the N for its end-to-end payoff (0.9975^20 = 0.951, 0.975^20 = 0.603 at N = 20).
Counts and the interval. Enter a step as eval counts (c of n) and the point estimate is c/n, with a Wilson score interval at your chosen confidence. Raising the per-step interval bounds to the N (uniform) or multiplying the per-step bounds across steps (list) carries the interval through to the end-to-end estimate. That band is an interval on the reliability estimate under the same independent-and-uniform assumption, not an exact joint confidence region. The Wilson interval is used because the Wald normal approximation is unreliable near 0 and 1 and at small n.
Honest limits. Every figure is deterministic arithmetic on the inputs you enter. The independence-and-uniformity assumption is a modelling choice; a real system with shared state fails in correlated clusters, which changes what a failure costs more than it changes the all-steps-clean rate. The topology-aware playbook works through that in reliable AI agents in production.
Formula: end-to-end = p^N · required per-step = target^(1/N) · list = product of per-step rates
Questions
What is the end-to-end reliability of a 20-step agent chain at 0.95 per step?
At 0.95 reliability per step across 20 independent steps, the chain succeeds end to end about 35.8% of the time, because reliability compounds as 0.95^20 = 0.358. A step that looks shippable at 95% in isolation yields a roughly one-in-three end-to-end success rate once twenty of them run in series. Raising each step to 0.99 lifts the chain to 0.99^20 = 0.818.
How do I set a per-step reliability budget for a target success rate?
Invert the compounding curve: the per-step reliability a uniform chain needs is target^(1/N). To hold a 20-step chain at 90% end to end, every step must clear 0.90^(1/20) = 0.9947; a 99% target demands 0.99^(1/20) = 0.9995 per step. Solve mode computes this budget and, given a current per-step rate, sizes the shortfall to target.
Which step should I harden first?
Harden the weakest step first. Because end-to-end reliability is the product of the per-step rates, the lowest rate leaks the most, so a retry or fallback there buys the largest end-to-end gain per unit of effort. Paste your per-step list and the calculator flags the weakest link. The reliability playbook covers which lever fits which failure.
Sources
- Assessing Product Reliability: the series-system model, where system reliability is the product of the component reliabilities
- Probable Inference, the Law of Succession, and Statistical Inference (Wilson score interval)
- Interval Estimation for a Binomial Proportion (why the Wilson interval beats the Wald normal approximation)
- Confidence Intervals for a Proportion