Retry storm (agent systems)

Retry storm is the failure mode where correlated retries turn a brief downstream fault into a sustained one. Several layers of an agent stack reissue the same failing call at once, the attempts multiply as a product across those layers rather than adding up, and the load they generate holds the downstream saturated after the original trigger has cleared. It describes the traffic instead of any one agent’s behavior, which is why the component that started it looks healthy in its own logs.

Two things change when the clients doing the retrying are agents. A retried agent step is rarely one HTTP call, since it replays a whole tool loop with the accumulated context attached, so the amplification lands in tokens as well as in requests. Many agent faults are also semantic, meaning the model returned something well formed and wrong, and a second attempt reproduces that at close to the original probability, adding load without adding much chance of success.

A retry is not even guaranteed to re-run the same computation. Anthropic’s server-side fallback parameter re-issues a refused request to a different model inside a single API call, either one the caller names or, in the default mode it shipped in July 2026, one the provider picks by refusal category. Our pre-registered routing study saw that path fire on 20 of 28 low-effort calls, with every rescued answer produced by the cheaper Claude Opus 4.8 instead of the Claude Fable 5 the caller had asked for, and the run behind the refusal tax holds the method and the intervals.

Retry storm sits beside error propagation in the containment cluster, and the two travel by different roads. Propagation moves through the semantics of the work, which is why propagation radius is read across the graph of who consumes whose output. A storm moves through the shared resource, so it reaches agents that never touched the faulty output and were only unlucky enough to want the same rate limit.

Count attempts rather than requests, or the storm stays invisible in your own telemetry.

How to calculate retry amplification

Retry amplification is the quantity to compute: attempts observed at the downstream divided by distinct originating requests, over one stated window. Count the attempts where they land, since the multiplication forms downstream. Google’s SRE book works the arithmetic in addressing cascading failures: if the backend, frontend and JavaScript layers “all issue 3 retries (4 attempts), then a single user action may create 64 attempts (4^3) on the database” (Beyer et al., 2016). An agent stack layers the same way, from the provider SDK through the framework’s tool-call wrapper to the orchestrator that re-dispatches a failed sub-agent; the system reliability calculator runs that chain product in the direction of reliabilities instead of attempts.

Report the factor with the window it covers, the denominator of distinct originating requests, and the configured retry count at every layer you can see, since those counts are the terms the product is built from. The share of retries that actually succeeded belongs beside it, a proportion that needs an interval like any other, which the pass-rate interval calculator will bracket. The flagship profiler, still unshipped, is designed to inject a bounded downstream failure and record every attempt that follows, which is why no amplification factor of ours appears on this page. The retry path we have measured is the provider’s own: a pre-registered routing run caught refused requests being re-served by a cheaper model inside a single API call, an attempt the client never issued and never counts. Instrument at the client holding the API credential, tagging every attempt with the id of the request that spawned it.

A timeout set below the downstream’s real tail latency manufactures storms on its own: the client abandons a request the server is still working on, issues another, and the server pays twice for one answer. Telling a stall, a hang and an ignored stop signal apart is its own diagnosis, which why agent runs fail to terminate works through. Attempts abandoned that way also drop out of the denominator of whatever pass rate you publish next, the distortion answer coverage exists to expose.

Retry storm vs backpressure

Backpressure is a signal that travels upstream: a component that cannot keep up tells its callers to slow down, and the callers are built to obey. In a bounded queue it is a buffer that blocks the producer once it fills; in an agent pipeline it is an orchestrator that stops dispatching sub-tasks while the worker pool is saturated. A retry storm is what the same system does once that signal is absent or ignored.

Which of the two you have depends on whether the slow-down signal exists at all. Where no such signal exists, a downstream that has merely got slower looks to its callers exactly like one that has failed, and trying again is the locally correct response to a failure. That is how most storms begin, out of a decision that was right in the small. Where the signal exists and no retry policy sits beside it, a one-off blip that a single delayed attempt would have cleared becomes a user-visible failure. Amplification tells you whether attempts are multiplying, and the share of dispatches a slow-down signal deferred tells you whether the brake is connected at all. The signaling side gets its own treatment under backpressure in agent pipelines, and cutting calls to a failing dependency once a threshold trips is the blunter version of the same brake.

Retry storm vs thundering herd

Thundering herd names a synchronization problem: many clients waiting on the same event wake at the same instant and hit one resource together. The textbook case is a set of processes blocked on a single socket, all woken by one incoming connection, and the caching version is a hot key expiring while every request that wanted it goes to the origin at once. Correlation in time is the whole mechanism there, and one round of it is enough to hurt.

A retry storm carries a feedback loop that a herd lacks. Each round of failures generates the next round of attempts, so the load climbs for as long as the fault persists. The two share a cure and a trap: jitter breaks the time correlation in both cases, and on its own it leaves the multiplication untouched, since spreading 64 attempts across a minute still sends 64 attempts. Peak concurrent attempts is the herd’s number, and attempts per originating request is the storm’s.

The SRE remedies assume a fault a later attempt can fix

Retry storm reaches this lane from site reliability engineering, where the dynamic is well documented and the remedies are settled: exponential backoff with jitter, a retry budget capping how many retries a process may issue (the SRE book’s worked example allows 60 a minute, then fails the request outright), and a breaker at the client. Those remedies are legitimate and most of them port. They do assume the fault is transient in the ordinary sense, meaning the identical call succeeds later because the resource recovered.

Agent stacks strain that assumption at two points. A malformed tool argument, a prompt that trips a guardrail, and a plan the model regenerates the same way are all faults the next attempt reproduces, so backoff buys time that nothing uses. The downstream is also a metered API you do not operate: Anthropic’s rate-limit documentation returns 429 with a retry-after header, states that earlier retries will fail, and warns that a sharp increase in usage can trip an acceleration limit by itself (as of 2026-08). Retrying harder against a token bucket lengthens the outage you are trying to end. Which of these patterns survive the move to agents is argued in the prior-art audit on microservices resilience.

Amplification says how much traffic a fault generated. Containment rate says how far the fault itself traveled. A system can contain a fault perfectly at the first hop and still lose the run, when the attempts spent on that one contained fault exhaust the budget every other request was sharing. How a single wrong output becomes a system-level failure is covered in our reframe of OWASP’s ASI08 as reliability engineering, and retry as one line item spent against a reliability target is in budgeting for failure in production. Track attempts per originating request from the first day of a deployment, because that ratio cannot be recovered later out of logs that counted only requests.