LatentEval

INSTRUMENT | Propagation & containment

Multi-Step Agent Reliability Calculator (Independent Steps)

4 cited sources

Compute the end-to-end success rate of a multi-step agent chain from its per-step reliability, and the per-step budget your target reliability demands.

How reliable is a multi-step agent chain? Compounding errors settle it: twenty steps at 0.95 each succeed end to end only 35.8% of the time. Invert that for the per-step reliability budget a target demands, which feeds cost per success, failure cost, and the SLO to clear. Every figure assumes steps fail independently; when they share a cause, use the correlated cascade simulator. Built on what a reliability budget buys in production and how a single fault travels a chain.

What do you want to compute?

A decimal like 0.95, a percent like 95%, or a bare 95. Anything above 1 is read as a percent, so 1.5 means 1.5%.

Check this value.

Enter this step as eval counts (c of n) for a confidence interval

Check this value.

Check this value.

How many steps run in series.

Check this value.

Only affects the interval when you enter eval counts.

Check this value.

35.8%

0.95^20 = 0.358 end to end over 20 steps

What one lever, applied to every step, buys.

ApproachEffective per-stepEnd-to-end
Export

end-to-end = p^N; required per-step = target^(1/N); list = product of per-step ratesHow?

How this is calculated

The model treats a workflow as steps running in series: the end-to-end result lands only when every step holds. Give each step an independent success probability p, and a chain of N steps succeeds end to end with probability p raised to the N. That is the sequential-chain worst case, and it is the assumption baked into every number here.

The name for the rule. Reliability engineering calls this the series-system model: the NIST series-system entry cited below gives system reliability as the product of the component reliabilities, which is the form used here. A chain is therefore at most as reliable as its weakest step, since multiplying by every other rate can only pull the product down. Older reliability literature also carries the rule under the name Lusser's Law. In an agent chain that product is error propagation in its simplest shape, where one step's failure ends the run and no later step recovers it. A system whose later steps repair a bad intermediate output has broken the series assumption, and the figure here is a floor for it rather than an estimate.

Worked example. At 0.95 per step over 20 steps, 0.95^20 = 0.358, so a chain of twenty 95%-reliable steps succeeds end to end about 35.8% of the time. Push each step to 0.99 and 0.99^20 = 0.818; let it slip to 0.90 and 0.90^20 = 0.122.

Solving for the budget. Invert the curve and it becomes a budget. To hold a twenty-step chain to a 90% end-to-end target, every step must clear 0.90^(1/20) = 0.9947; a 99% target demands 0.99^(1/20) = 0.9995 per step. That is the per-step reliability budget the target forces.

A heterogeneous list. The per-step list multiplies a mixed chain: end-to-end is the product of the entered rates, and the lowest rate is flagged as the weakest link, where one lever buys the most.

What a lever buys. Uniform mode also shows what one lever, applied to every step, buys. One retry raises a step to 1-(1-p)^2 = 0.9975; two retries to 1-(1-p)^3 = 0.999875; a fallback that recovers half of a step's failures to 1-(1-p)(1-0.5) = 0.975. Each effective per-step rate is raised to the N for its end-to-end payoff (0.9975^20 = 0.951, 0.975^20 = 0.603 at N = 20).

Counts and the interval. Enter a step as eval counts (c of n) and the point estimate is c/n, with a Wilson score interval at your chosen confidence. Raising the per-step interval bounds to the N (uniform) or multiplying the per-step bounds across steps (list) carries the interval through to the end-to-end estimate. That band is an interval on the reliability estimate under the same independent-and-uniform assumption, not an exact joint confidence region. The Wilson interval is used because the Wald normal approximation is unreliable near 0 and 1 and at small n.

Honest limits. Every figure is deterministic arithmetic on the inputs you enter. The independence-and-uniformity assumption is a modeling choice; a real system with shared state fails in correlated clusters, which changes what a failure costs more than it changes the all-steps-clean rate. The topology-aware playbook works through that in reliable AI agents in production. To put a number on that shared cause rather than assume it away, run the same chain through the shared-failure version of this calculator, which reports both figures side by side.

Formula: end-to-end = p^N; required per-step = target^(1/N); list = product of per-step rates

Questions

What is the end-to-end reliability of a 20-step agent chain at 0.95 per step?

At 0.95 reliability per step across 20 independent steps, the chain succeeds end to end about 35.8% of the time, because reliability compounds as 0.95^20 = 0.358. A step that looks shippable at 95% in isolation yields a roughly one-in-three end-to-end success rate once twenty of them run in series. Raising each step to 0.99 lifts the chain to 0.99^20 = 0.818.

If each step is 98% reliable, how reliable is the whole chain?

Multiply the per-step rate by itself once per step: 0.98^N. Five steps at 98% hold together 0.98^5 = 90.4% of the time, ten steps 0.98^10 = 81.7%, twenty steps 0.98^20 = 66.8%, and fifty steps 0.98^50 = 36.4%. Quote N alongside the answer, because the step count does most of the work: fifty steps at 98% land within a point of twenty steps at 95% (0.95^20 = 35.8%). A two-point per-step failure rate is cheap in a short chain and decisive in a long one.

How do I set a per-step reliability budget for a target success rate?

Invert the compounding curve: the per-step reliability a uniform chain needs is target^(1/N). To hold a 20-step chain at 90% end to end, every step must clear 0.90^(1/20) = 0.9947; a 99% target demands 0.99^(1/20) = 0.9995 per step. Solve mode computes this budget and, given a current per-step rate, sizes the shortfall to target.

Which step should I harden first?

Harden the weakest step first. Because end-to-end reliability is the product of the per-step rates, the lowest rate leaks the most, so a retry or fallback there buys the largest end-to-end gain per unit of effort. Paste your per-step list and the calculator flags the weakest link. The reliability playbook covers which lever fits which failure.

Sources

  1. Assessing Product Reliability: the series-system model, where system reliability is the product of the component reliabilitiesNIST/SEMATECH e-Handbook of Statistical Methods Retrieved
  2. Probable Inference, the Law of Succession, and Statistical Inference (Wilson score interval)Journal of the American Statistical Association (Wilson, 1927) Retrieved
  3. Interval Estimation for a Binomial Proportion (why the Wilson interval beats the Wald normal approximation)Statistical Science (Brown, Cai, DasGupta, 2001) Retrieved
  4. Confidence Intervals for a ProportionNIST/SEMATECH e-Handbook of Statistical Methods Retrieved