Backpressure (agent pipelines)

Backpressure is the signal a saturated stage in an agent pipeline sends back to whatever feeds it, asking the producer to slow down so the queue between them stays bounded. A pipeline that carries the signal refuses work at its entrance, where a refusal is cheap and visible. One that swallows it fills a queue nobody watches, and the symptom surfacing later is a run that stalls without ever failing.

Bounding the work in flight also bounds how many downstream steps a wrong result reaches before the pipeline drains, which is what propagation radius counts. Circuit breaking removes a failing dependency and bulkhead isolation stops one stage draining a shared pool, while backpressure keeps calls flowing at the rate the downstream can absorb. Whether the three port cleanly onto agent graphs is settled in our audit of the microservices resilience patterns, which files backpressure under the ones needing translation.

How to measure backpressure in an agent pipeline

Count offered items and admitted items at the admission point over a fixed window, then divide the second by the first. Report a high-percentile queue wait alongside that share, because a pipeline can admit everything it is offered and still run an hour behind its producer. Put an interval on the share the way you would on a pass rate: the Wilson and Clopper-Pearson calculator takes k of n directly, on the reasoning under eval confidence interval.

The admission point is usually the orchestrator’s dispatch loop or the gateway brokering every tool call, and part of the signal already arrives there. The Claude API returns a 429 with a retry-after header and anthropic-ratelimit-*-remaining counters showing how much budget is left, so a dispatcher can slow down ahead of the wall. The same documentation notes that a 60-requests-per-minute limit may be enforced as 1 request per second, so a burst averaging comfortably under the ceiling still trips it.

Counting items is the wrong unit for this measurement. Downstream capacity for a model stage is denominated in tokens per minute, so ten queued items can sit inside budget or far outside it depending on the context each drags along. A large tool return moves that answer between queueing and sending. Meter the queue in tokens, which also makes a tool return clipped at the framework’s default limit and budget saturation legible as one event.

Load shedding changes every number measured downstream

Because the mechanism defers or refuses work, it splits traffic into an offered rate and an admitted rate, and every number computed downstream rests on the admitted subset alone. The admitted share is a proportion carrying an interval, and it belongs beside any pass rate from the same window, which is the denominator discipline answer coverage enforces.

Shedding is never neutral about what it drops. A gate shedding by cost drops the long-context items first, and those are usually the hard ones. A gate shedding by arrival order drops whatever the retry queue was not holding at that moment. Either way the rate you publish rests on a subset whose membership correlates with difficulty, which is the distortion coverage conditioning names.

A system that quietly sheds its hardest requests posts a better pass rate than one that admits them and fails some. No line in the eval output separates the two. The repair costs one field in the dispatch log: record every shed item with its reason, score it as unresolved for that window, and publish the admitted share next to the rate. A stage that sheds by returning less rather than nothing owes the same accounting, since a degraded answer scored as a pass moves the number the way a dropped request does.

Backpressure vs rate limiting

Rate limiting is a policy the downstream side enforces on its own behalf. A server decides how much traffic it will accept from a caller in a window and rejects the excess, usually with a 429 and a hint about when to come back. Backpressure is the upstream response, where the caller reads that rejection, or reads the remaining-budget counters before any rejection lands, and lowers its own send rate.

Which half is missing decides what the gap costs you. A caller can sit behind a rate limiter with no backpressure at all, sending at full tilt and absorbing the rejections, which spends the downstream’s capacity on refusing traffic. Google’s SRE book describes a backend overloaded while most of its CPU goes into rejecting requests, and prescribes client-side throttling so requests above a self-imposed cap fail locally without reaching the network. A pipeline can also need backpressure where no rate limiter exists, at a validator, a human review queue, or a checkpointing step, none of which returns a 429. Track the rejection rate at the boundary and the send rate above it, since only the pair shows whether your dispatcher responds to a signal it already receives.

Backpressure vs retry storm

A retry storm is the failure where many callers retry a struggling dependency at once, so offered load multiplies at the moment the dependency has least capacity to serve it. Backpressure keeps one from starting, since a dispatcher lowering its send rate on the first signal never assembles the correlated burst. Both live in the same piece of code: the policy a caller runs after a failed call.

Tuning one and ignoring the other leaves a retry policy fighting its own admission gate. Backpressure with no retry budget still leaks, because a policy ignoring retry-after re-offers the same work while the bucket is empty. Retries with no backpressure behave worse here than in the microservices literature they came from, since many agent faults are semantic: re-sending an identical prompt reproduces the same wrong answer and feeds it downstream as trusted input again. Google’s SRE guidance caps retries at three per request, four attempts including the original, and holds a client’s retries below 10% of its traffic, both reasonable starting points, though a semantic fault often deserves no retry at all.

Backpressure in stream processing means something narrower

In Reactive Streams, Kafka Streams and Flink, backpressure has a tighter definition: a consumer signals demand for a bounded number of elements, the producer may not exceed it, and the queue between them stays bounded by construction. The Reactive Streams specification requires that signaling be non-blocking and asynchronous, since a synchronous back-channel would negate the benefit of asynchronous processing, and it leaves the carrying mechanism to each implementation. That meaning is the established one, and this page does not displace it.

The demand protocol does not survive the move unchanged. Elements in a stream cost roughly what their neighbors cost, while an agent step’s cost varies by orders of magnitude with context length and tool depth, so a demand count in items bounds nothing about resource use. Staleness breaks it again: a stream element that waits is still valid when it is finally processed, and a queued agent item goes off once the state its plan was computed against moves on. The capacity assumption breaks hardest of the three. Reactive Streams assumes a consumer that knows its own limit, whereas a model endpoint’s is set by the provider, applied at the organization level, and shared with every other workload on the account.

One construction does survive, with the shape kept and the units changed: express demand in tokens per window rather than items, attach a deadline to each queued item so a stale one is dropped, and re-read the provider’s counters every window.

Where the admission gate sits is settled by the wiring above it, and the orchestration pattern you already chose fixed how many stages can saturate at once. Backpressure sits alongside retry, fallback and checkpointing as levers spent against a target end-to-end rate, a budgeting view worked through in budgeting for failure in production, on arithmetic the system reliability calculator runs for you.