Retry, Verification, and Fallback Reliability Optimizer for LLM Agents

Every attempt on the primary path ends one of three ways: correct with probability p, a detected failure (an error, a timeout, an invalid output) with probability q = (1 - p) · d, or a silent wrong answer with probability s = (1 - p) · (1 - d), where d is the share of failures that surface on their own. That split is what separates the policies. A retry or a fallback can only act on a detected failure; a silent wrong answer never triggers one. A verifier turns silent failures into detected ones, catching a wrong answer with probability cr and wrongly rejecting a correct one with probability fr. A vote acts on silent failures by redundancy. Attempts are treated as independent and identically distributed throughout. Most agent faults are semantic rather than transient, which is what loads the silent branch; that finding, and the lever table this tool puts a price on, come from the levers that hold up in production agent systems.

The six policies. Single run ships whatever one attempt produced. Retry re-runs the primary on a detected failure, up to n attempts, and ships the first answer produced: with G = (1 - q^n) / (1 - q) the expected attempts, it ends correct with probability p · G, wrong with s · G, and unanswered with q^n. Fallback runs the primary once and, on a detected failure, a different path once at its own rate pf: correct p + q · pf. Majority vote runs k attempts and ships the answer held by more than k/2 of them: correct with the binomial tail P(Bin(k, p) > k/2), wrong with P(Bin(k, s) > k/2) if wrong answers coincide (the worst case, so an upper bound), otherwise no answer. Verifier-gated retry checks each produced answer and re-runs on a reject: per attempt it accepts a correct answer with probability p(1 - fr), accepts a wrong one with s(1 - cr), and re-rolls with r = q + p · fr + s · cr, the chance an attempt ends in a detected failure, a false reject, or a caught wrong answer; over the loop it ends correct with p(1 - fr) · G(r), wrong with s(1 - cr) · G(r), and unanswered with r^n. Escalation runs the same loop and hands the r^n that remains to a resolver assumed to answer correctly.

Costs. Run cost is an expected value: primary attempts times the attempt cost, verifier checks (only produced answers are checked) times the check cost, plus the fallback or escalation charge times the probability it fires. Expected total cost adds the wrong-answer cost times the probability a wrong answer ships and the no-answer cost times the probability nothing ships. The recommendation is the policy with the lowest expected total; ties break toward the higher delivered-correct rate. Only ratios between the costs move the recommendation, so any unit works as long as every cost uses it. The sensitivity table re-derives the recommendation at 0.25x, 0.5x, 2x, and 4x of the entered wrong-answer cost, the softest input.

Worked example, the preloaded inputs. A primary path at 0.80 with half of its failures detected gives q = 0.10 and s = 0.10 per attempt. Retry with up to three attempts lifts correct delivery from 80% to 88.8% (0.8 × 1.11 expected attempts) but also lifts wrong delivery from 10% to 11.1%, because retries convert detected failures into more answers of the same mix. A fallback at 0.90 buys 89.0% correct at 1.4 per task. Majority vote of three reaches 89.6% correct with 2.8% wrong at a flat cost of 3. The verifier (85% catch, 10% false reject, 0.25 per check) rejects 16.5% of attempts and never sees the 10% that already failed loudly, so 26.5% of attempts end in a re-roll; over up to three attempts the loop ends 96.1% of tasks correct and 2.0% wrong at 1.636 per task, and with a wrong answer priced at 20 and no answer at 5 its expected total, 2.129, is the lowest of the six. Escalating the 1.9% left over at 15 each raises correct delivery to 98.0% at a total of 2.315.

How this differs from pass@k and the chain calculator. pass@k and reliability@k describe repeated runs of the same task with no decision rule and no cost: at least one of k passes, or all k pass. This tool prices a policy that decides which answer ships, at what cost, and how the task ends when nothing acceptable is produced. It works on one task or one step. The lever rows in the multi-step agent reliability calculator are these same retry and fallback levers at d = 1, where every failure announces itself; the detected share, the verifier, the vote, and the cost axis are what this tool adds. To see what a lever buys across a multi-step chain, carry the policy's delivered-correct rate into that calculator as the effective per-step rate.

Honest limits. Independence is the assumption everything rests on. A task the primary path gets wrong every time, or an outage that fails every attempt at once, makes retry and voting buy less than these figures say. How much a retry can still buy when a share of failures traces to one cause shared across steps is what the shared-failure simulator puts numbers on. The fallback's real advantage is the one thing the model cannot see, a second path that does not share the primary's failure causes, so read the fallback row as a floor. Verifier errors compound with the attempt cap: false rejects can turn correct answers into no answer, which is why the verifier loop can deliver fewer correct answers than a plain retry at low n. The vote's wrong-answer figure assumes wrong answers agree; scattered wrong answers push those cases toward no answer instead, which makes the vote's total cost optimistic whenever a missing answer costs more than a wrong one. Costs are expected values per task, not bounds on any single task. Where retry helps and where it re-runs a semantic fault is worked through in cascading failures in agent systems.

Formula: expected total cost = run cost + P(wrong) · wrong-answer cost + P(no answer) · no-answer cost

See Sources below