Agentic RAG architecture, and where each pattern breaks

You gave your RAG pipeline an agent, and now it retrieves four times for a question that used to take one, occasionally loops without converging, and when the first retrieval brings back the wrong passage every later step reasons confidently from it. The decision in front of you is whether that agent earns the latency and the new failure surface it introduced, or whether a single retrieve-and-read was already doing the job. Autonomy buys recovery paths. It also gives an error more places to start and more hops to travel before anything catches it.

Reach for agentic RAG when a query needs multi-step reasoning or an adaptive retrieval strategy, and treat every extra hop as both a recovery path and a new failure surface. The architecture is classic RAG plus three parts: a planner or router that decides the retrieval strategy, a tool layer that runs the retrievals, and a self-correction loop that critiques and retries. Its reliability turns on one quantity the architecture diagrams rarely label: what fraction of a hop’s errors the next step contains before they reach the answer. Skip that quantity and you ship a system that is more capable and less predictable at once, which is the hardest combination to debug in production.

What is agentic RAG?

Agentic RAG is retrieval-augmented generation in which one or more autonomous LLM agents control the retrieval process instead of running it once in a fixed order. The agent decides whether to retrieve, rewrites the query, chooses a source or tool, judges whether the returned context is sufficient, and loops until it is, applying the agentic patterns of reflection, planning, and tool use to the retrieval step itself.

That definition follows the 2025 preprint survey by Singh and colleagues, Agentic Retrieval-Augmented Generation, which frames the design as embedding autonomous agents into the pipeline to escape “static workflows” that “lack the adaptability required for multi-step reasoning.” Classic RAG, in the original formulation by Lewis and colleagues (NeurIPS 2020), pairs a parametric language model with a non-parametric index and retrieves once. Agentic RAG keeps that machinery and puts a decision-maker in front of it.

Classic RAG vs agentic RAG: what the agent adds

Classic RAG is a function: query in, one retrieval, one answer out. Agentic RAG is a loop that can retrieve zero times or ten, rewrite the question, and stop when a self-check passes. The table below reads each added degree of freedom against what it costs you in reliability, because every capability the agent gains is also a new way for a run to diverge.

TABLEShow full table (6 rows)Showing full table (6 rows)

Dimension	Classic RAG	Agentic RAG	Reliability consequence
Retrieval trigger	Always retrieves once	Agent decides whether and when to retrieve	Skips needless lookups, but can over-retrieve or loop without converging
Query handling	User query used as-is	Agent rewrites and decomposes the query	Recovers from bad phrasing, but can drift off the user’s intent
Control flow	Fixed pipeline	Dynamic loop with a stopping condition	Adapts to hard queries, but the stop rule can fire early or never
Retrieval hops	One	Many, across sub-questions	Answers compositional questions, but multiplies per-hop failure
Error handling	None; a wrong chunk becomes a wrong answer	Self-critique and re-retrieve (Self-RAG, CRAG)	A containment gate for retrieval faults, when it fires
Cost and latency	One retrieval, one generation	Several of each, variable per query	Higher and less predictable; a tail of slow, expensive runs

The table is a decision aid; neither column wins outright. The loop is worth its cost when the answer depends on what an earlier step found, and it is a liability when the answer does not. A single-hop factoid lookup gains nothing from an agent except variance and latency, a point the when-not-to-use section turns into a rule.

Inside a RAG agent architecture: planner, tools, memory, and the self-correction loop

The three added parts, plus the memory they read and write between hops, give every RAG agent four working parts. Read a RAG agent architecture as a loop over them: query, then planner/router, then the tool layer, then memory, then a critic, then the answer, with each part carrying its own reliability concern.

The router is the same routing problem covered in routing models by task, and it fails the same way. The tool layer is why a stateless, well-scoped tool surface matters for reproducibility.

Read the architecture as four things the agent can do that a pipeline cannot: choose, check, chain, and delegate. Each one is a pattern with its own failure surface.

Four patterns, four failure surfaces

Each move starts its errors in a different place, and naming the move tells you where.

The planner/router pattern (choose). A single agent reasons about the query and acts on it, in the ReAct style of interleaving a reasoning trace with retrieval actions. The paper that introduced it, ReAct (Yao and colleagues, ICLR 2023), reports that interleaving reasoning with retrieval “overcomes issues of hallucination and error propagation prevalent in chain-of-thought reasoning” by grounding each step in an external source. A more recent router, Adaptive-RAG (Jeong and colleagues, NAACL 2024), makes the choice explicit: a classifier predicts a query’s complexity and routes it to no retrieval, single-step retrieval, or a multi-step loop. Newer work trains that routing policy instead of prompting it, as in Search-R1 (Jin and colleagues, 2025 preprint), which uses reinforcement learning to teach a model when to issue its own search queries mid-reasoning.

The self-corrective pattern (check). The agent judges its own retrieval or answer and retries when it falls short. Self-RAG (Asai and colleagues, ICLR 2024) trains a model to retrieve on demand and critique its output with reflection tokens, reporting better factuality and citation accuracy than standard retrieval-augmented models. Corrective RAG adds a lightweight retrieval evaluator that returns a confidence degree and, on low confidence, triggers a web search to replace weak context.

The multi-hop pattern (chain). For compositional questions, the agent retrieves, reasons, and retrieves again, each hop’s query shaped by the last hop’s finding. The peer-reviewed IRCoT method (Trivedi and colleagues, ACL 2023) reports that, run with GPT-3, it improves retrieval by up to 21 points and downstream QA by up to 15 points across four multi-hop datasets while reducing hallucination.

The orchestrated pattern (delegate). For queries that span sources, a coordinator agent splits the work across specialized retrieval agents, one per source or method, and merges their returns. The survey’s taxonomy calls this the multi-agent variant, in which a master retrieval agent delegates sub-queries to agents specialized in structured search, semantic search, or the web. This is the topology the multi-agent failure taxonomy MAST was written for. MAST, the first empirical taxonomy of multi-agent LLM failures (14 modes in 3 categories, Cemri and colleagues, 2025 preprint), names inter-agent misalignment as one of its three categories: faults that cross the seam between one agent’s return and the step that trusts it.

Each pattern maps to where its error starts, how it travels, and the containment gate that holds it.

TABLEShow full table (4 rows)Showing full table (4 rows)

Pattern	Where the error starts	How it travels across hops	The gate that holds it
Choose (planner/router)	A mis-classified query picks the wrong tool or hop count	Every downstream step inherits the wrong premise	A route-confidence threshold that falls back to the fixed single-retrieval pipeline
Check (self-corrective)	A mis-calibrated critic passes bad context or rejects good context	The loop ships the error, or never converges	The critic itself, calibrated against known-bad retrievals
Chain (multi-hop)	The first wrong hop is written to memory as fact	Each later hop reasons from the poisoned finding	A per-hop sufficiency check before a finding is committed to memory
Delegate (orchestrated)	A sub-query is misrouted or answered wrong at the hand-off	The merge folds the bad return in as one trusted source among several	The orchestrator verifying each sub-agent’s return before it merges

The verdict: the router needs a fallback, the critic needs calibration, the chain needs a check between hops, and the orchestrator needs to verify each return before it merges. Three of the four gates are things you add, so a design that draws the pattern without drawing its gate has skipped the reliability half.

Where agentic RAG breaks: the error propagates across hops

The reliability lens on agentic RAG is a single dynamic. Adding hops multiplies failure, and the self-correction loop is the only thing that divides it back down. This is the error-propagation chain worked through in our analysis of cascading failures in agent systems, applied to a retrieval loop.

How hops multiply failure

Take the multiplication first, on deliberately round numbers. Suppose each hop of a four-hop chain succeeds independently with probability 0.90, and nothing catches a failure. End-to-end success is 0.90 to the fourth, about 0.66. The same 0.90 hop that looks reliable in isolation ships a wrong final answer a third of the time once you chain four of them. These are schematic inputs chosen to show the shape, not measured rates. The shape is the point: a per-hop reliability that reads as fine compounds into an end-to-end reliability that does not.

Two coined quantities from the lane make that concrete. How far an early error travels before it surfaces is its propagation radius; how many hops deep the chain runs before a check fires is its propagation depth. A pure multi-hop chain with no critic has an unbounded radius, because the first wrong retrieval becomes memory and every later hop reasons from it as fact. That is the single-agent echo of the inter-agent misalignment MAST names for multi-agent systems: a fault that crosses the seam between one step and the next that trusts it.

How the critic divides it back down

Now the division. The self-correction loop is a containment gate, and its value is the fraction of hop-level errors it stops before they reach the answer. A critic that catches half of bad retrievals before they enter memory turns the unbounded chain into a bounded one, which is the property the lane calls cascade-resistance. So the reliability question for any agentic RAG design reduces to two numbers per hop: how often the hop fails, and what fraction of those failures the next check contains. A diagram that shows the loop but never labels the containment rate is describing the capability and hiding the risk.

The reliability of an agentic RAG system is set less by any single hop’s accuracy than by whether a wrong hop is caught before it becomes the next hop’s premise. Measure the gate before you trust the generator.

When not to reach for agentic RAG

The default for most retrieval problems is still classic RAG, and agentic RAG has to argue its way past that default on each use case. Reach for the loop only when a query clears one of these bars.

• The question is compositional. The answer depends on a fact you can only retrieve after retrieving an earlier fact. Multi-hop reasoning is the case agentic RAG was built for.
• Retrieval strategy must adapt. Different queries need different sources, tools, or hop counts, and a fixed pipeline would serve some of them badly.
• A wrong answer is costly enough to pay for a check. The self-correction gate earns its latency when a confidently-wrong answer does real damage.

Skip the agent when the query is a single-hop factoid, when latency is tight, or when your corpus is small and clean enough that one retrieval reliably holds the answer. Extra hops also carry a quieter tax: each one adds context, and models under-use information buried in the middle of a long context (Liu and colleagues, peer-reviewed), so a longer chain can bury the very passage it worked to find. And every RAG system inherits the finding from the seven-failure-points experience report (Barnett and colleagues, peer-reviewed) that “validation of a RAG system is only feasible during operation,” which an agentic loop makes harder still by widening the space of paths a query can take.

How to evaluate an agentic RAG system

You evaluate RAG agents the way you evaluate any RAG pipeline, then add the loop’s own failure surface on top. The base method is to score retrieval and generation separately, put a confidence interval on each metric, and attribute each failure to the stage that caused it, the discipline in how to evaluate a RAG pipeline beyond a single score. The separable-dimensions framing comes from the peer-reviewed RAGAS work (Es and colleagues, EACL 2024), covered in evaluating RAG with RAGAS.

The agentic part adds three requirements:

• Attribute failures per hop, not per pipeline, so you know which retrieval in the chain broke; this is stage-level failure attribution extended along the loop.
• Score the whole loop for consistency across repeated runs, because a system that answers correctly on one run and wrongly on the next is unreliable even at a good average; the lane tracks that as reliability@k, and a difference between two builds is a paired significance question that overlapping bars cannot settle.
• Measure the self-correction gate directly, by injecting known-bad retrievals and recording the fraction the critic catches, which turns “the loop self-corrects” from a claim into a containment-rate number with an interval.

This site points toward a reliability profiler, and the agentic-RAG view it is designed to include would compute those per-hop pass rates and their intervals from your own eval runs. That is design intent; no score is measured yet, so this page quotes none, and every external number here traces back to a source you can open.

The decision rule for adding the agent

Agentic RAG is classic RAG with a decision-maker in front of it, and the decision-maker is worth its cost exactly when a query needs it and a hazard when it does not. Three moves make the choice defensible:

• Add the agent only for compositional or adaptive queries. For single-hop factoids, keep the fixed pipeline and spend the saved latency elsewhere.
• Design the containment gate before the hops. A loop without a calibrated self-check multiplies failure with nothing to divide it back down.
• Evaluate per hop and across runs. Attribute each failure to its hop, report reliability across k runs, and put an interval on the gate’s catch rate before you trust it.

Build the loop that way and the extra hops buy reasoning you could not get from one retrieval. Build it without the gate and you have shipped a more capable system that fails in more places at once. The reliability glossary and the analysis index hold the measurement vocabulary this architecture leans on.