Failure domain (agent topology)

Failure domain, in an agent topology, is the set of agents, tools and steps that a single fault can reach given the way the system is wired. It is a structural quantity read off the graph, which separates it from propagation radius: a domain is the boundary a fault could cross, and a radius is how far one observed fault actually traveled before something stopped it. You can draw a domain before anything runs, and a radius only becomes available afterwards.

Upstream of the term sits error propagation, the mechanism by which one agent’s output becomes another agent’s trusted input, and the domain is the population that mechanism can recruit. Downstream sit the measurements that say how much of the domain a real fault used: containment rate at each boundary, propagation depth along the chain, and cascade resistance once a whole wiring is scored on one axis. Draw the domain first, because every containment number collected afterwards is a claim about where its edge sits.

How to draw and size a failure domain

Pick the origin and the fault class, walk the graph forward over every edge that would carry that class of fault, and take the closure of the walk. Two kinds of edge carry faults, and only one of them shows up in an architecture diagram. Message edges carry a fault from the producer of a wrong output to whatever consumes that output. Shared-resource edges carry it sideways: a scratchpad two agents both read, a vector index one writes and several query, a tool credential a whole fleet holds. Stop the walk at any edge where a check can reject that specific fault class, since a validator that catches malformed JSON does nothing about a well-formed wrong number, and a boundary bounds only the faults it can recognize. The size of the domain is the count of distinct agents and tools inside the closure, given over the total. Composing per-step reliability across that many steps is what the system reliability calculator turns into an end-to-end figure.

Every agent that can read what a broken agent wrote is inside its domain, whether or not anyone drew the line.

Picture an orchestrator that fans work out to a set of workers and merges their returns. A wrong return from one worker lands at the hub, and once the hub redistributes merged context without validating it, every sibling sits inside that worker’s domain, which is the failure orchestrator-worker reliability scores. Instrument the drawing wherever the graph is already written down: the orchestrator’s routing table, the tool registry, the reader list on the memory store. Then test it rather than trusting it: seed a known fault at the origin under fault injection and record which nodes it touched. A fault that surfaces outside the set you drew is telling you about an edge you missed, and reconstructing how it got there is failure attribution.

Because a domain is a set rather than a rate, it carries no confidence interval, and two stated conditions do that work instead. Every domain is drawn relative to an origin, the node where the fault appears, and relative to a fault class. A corrupted tool return and a revoked credential can start at the same node and reach different sets, because they travel along different edges. So a domain is reported as an origin, a fault class, and the edge types you counted as carrying that class, with its size given over the whole agent population so that the denominator travels with the number.

Failure domain vs propagation radius

Propagation radius is breadth, measured: the count or fraction of downstream agents and steps one fault reached before containment, read across a specific topology and never quoted as a global figure. The informal phrase “blast radius” is reaching for that same quantity. A failure domain is the reachable set the measurement takes place inside, so the radius of any single fault is bounded above by the size of the domain that fault started in.

The drawn size and the measured radius often disagree, and what the gap means depends on which of the two is larger. A wide domain with a small measured radius means the boundaries held. Suppose nine agents could have received a bad context and a validator on the first handoff rejects it: the fault reaches one of the nine, and on that schematic system the domain read alone overstates exposure ninefold. A measured radius that lands outside the drawn domain means the drawing was wrong, and the missing edge is almost always a shared-resource one, since message edges are the ones people remember to draw. Run that way round often enough and the measurement becomes an audit of the boundary rather than a report on the fault.

A report carrying the drawn size without the measured radius has described a possibility and left the event unrecorded. The measured version belongs to the propagation radius entry, which owns the reporting rules for it; this page names the boundary that measurement happens inside.

Failure domain vs bulkhead isolation

Bulkhead isolation is a partitioning technique borrowed from ship design: split a shared resource, a connection pool, a worker set, a token budget, into compartments so that exhaustion in one cannot starve the others. A failure domain names the region a fault can reach, and a bulkhead is one of the instruments you install to make that region smaller.

Split the worker pool in two while both halves still read the same scratchpad, and the domain has not moved at all, because the fault was never traveling through the pool. A domain can also shrink with no partition anywhere in sight, when one validator at one handoff refuses the fault class outright. Choose between a partition, a circuit breaker and a validator by asking which of the three moves the boundary you actually drew.

Cloud fault domains follow the hardware, agent domains follow the state

Cloud infrastructure uses these words for a hardware boundary, and that meaning is older and better established than this one. Google’s Site Reliability Engineering book defines a failure domain as the set of components of a system that can become unavailable as a result of a single failure, with racks, datacenters and geographic regions among its examples. Microsoft attaches the term to a product: in the Azure availability-set documentation, fault domains “define the group of virtual machines that share a common power source and network switch”.

The hardware reading stands on its own, and the agent question sits alongside it, because the two kinds of boundary get drawn by different things. Cloud fault domains follow the shared physical infrastructure. Agent failure domains follow shared state and shared trust, so the two pictures stop matching quickly. Two agents in separate availability zones, on separate hardware, sharing no power supply, are in one failure domain the moment they read the same memory store or present the same tool credential. Consolidation has that shape too, which is why putting every tool behind a single broker is worth sizing before it is adopted.

Teams arriving with the cloud vocabulary usually want a count of how many failure domains their system has, and there isn’t one, because the answer moves with the fault class you name. A system with tidy compute partitions can still hold one enormous domain for anything that travels through shared memory. The count that means anything states the fault class, the origin and the edges counted, then reports one domain per class and puts the classes side by side. How much of the rest of the distributed-systems toolkit survives the move is the subject of our audit of microservices resilience patterns, which finds that most of them port, unequally.

A boundary is worth drawing only when something enforces it, and the enforcement (a validator at the handoff, a partition, a breaker, a human gate) is laid out hop by hop in how one plausible wrong output becomes a cascade. Draw the domains for the fault classes you expect, then spend on the hops where the domain comes out widest.