RAG evaluation metrics, and what each one certifies

A RAG evaluation board fills up quickly. Faithfulness, answer relevancy, context precision, context recall and some flavor of groundedness check all land on the same dashboard as soon as anyone decides the pipeline needs measuring, and every one of them can come back high on a run that just told a customer something false.

Each of those scores is a certificate over one property of one stage, computed over the requests that came back answered. Those five certificates compose into five narrow guarantees, and “the answer was right” sits outside all of them. Our own pre-registered study of model routing and the refusal tax is the run this page leans on for what that answered-only population does to a score.

Read each RAG metric as the single question it answers, then name the failure it stays silent about. Context precision and context recall grade retrieval from opposite ends. Faithfulness and groundedness grade the answer against two different reference points. Answer relevancy grades the fit between the question and the response and reaches no further than that. A board where every number is green and nobody can say what each number excludes will keep passing a stale index for as long as the index stays stale.

A metric certifies one property of one stage

“Certifies” is doing real work in that sentence. A RAG metric takes a set of scored runs, isolates one property, and returns a proportion. The property is scoped to a stage. Retrieval-side metrics look at what came back from the index; generation-side metrics look at what the model wrote on top of what came back. The survey literature organizes the field on close to that split, treating retrieval and generation as separately measurable subsystems with their own families of scores (Yu et al., 2024, arXiv preprint).

The population being scored matters as much as the property. Every rate on the board is computed over the requests that produced an answer, which is a smaller set than the requests you sent. In the refusal-tax run, the premium tier’s pass rate read 96.4% (27 of 28, 95% CI 82.3 to 99.4%) once a server-side rescue answered every declined call. Counted over everything sent, with refusals scored as failures, the same task set gave a raw pass of 17.9% (5 of 28, 95% CI 7.9 to 35.6%). The rescue fired on 20 of 28 low-effort calls, and every one of those rescued answers was served by the cheaper model underneath. Scores get computed on answers. Twenty of those 28 answers came from a model other than the one the routing named.

So each metric below carries two columns worth reading together: the question it answers, and the question it declines.

TABLEShow full table (6 rows)Showing full table (6 rows)

Metric	The one question it answers	What it stays silent about
Context precision	Of the chunks retrieved, what share were relevant?	Whether the chunk holding the answer was retrieved at all
Context recall	Of the passages needed, what share were retrieved?	Whether the generator then used them
Faithfulness	Are the answer’s claims supported by the retrieved context?	Whether that context was correct or current
Groundedness	Is the output traceable to an identified source?	Whether the source is authoritative for the claim
Answer relevancy	Does the response address the question that was asked?	Whether the response is true, or complete
Retrieval drift	Has retrieval quality moved since the last measurement?	Whether the index moved or the questions did

Read down the third column first. The second column is the one every metric definition already hands you.

Precision and recall grade retrieval from opposite ends

Context precision asks what fraction of the retrieved chunks were relevant to the question. Context recall asks what fraction of the passages needed to answer the question actually made it into the retrieved set. One is scored over what came back, the other over what should have come back. A pipeline that retrieves twenty chunks to be safe buys recall by spending precision, and a pipeline that retrieves two spends recall to buy precision, which is why the pair is reported together or reported uselessly.

Low precision costs more than tokens. Liu et al. measured a U-shaped position effect across multi-document question answering and key-value retrieval: performance ran highest when the relevant information sat at the beginning or the end of the input context, and degraded significantly when the model had to reach for it in the middle (Lost in the Middle, peer-reviewed, TACL volume 12, 2024). Padding the context window to protect recall can therefore bury the answer-bearing chunk in the region a model reads least reliably. The pillar on RAG failure modes files the precision loss under off-topic drift and the burial itself under silent context truncation, each with its own gate. The token arithmetic underneath a chunking plan belongs to the chunk-size calculator, which reports index size and context budget and will not tell you which chunk size retrieves better.

Whether a vector index is the right retrieval mechanism in the first place is still an open argument in 2026, and how agentic grep-style search measures up against embedding retrieval covers that claim and the point where it stops. Precision and recall stay the right instruments either way. Only the thing being graded changes.

Faithfulness scores the answer against its context, groundedness against a source

Faithfulness asks whether the claims in the answer are supported by the context retrieved for it. Groundedness asks whether the output is traceable to an identified source at all, retrieval or otherwise. The two words get used interchangeably, and then the two checks fail on different things.

Faithfulness takes the retrieved context as its standard of truth. Retrieve a document that is confidently wrong, or three years out of date, and a well-behaved generator will produce a faithful answer built on it, and the faithfulness score will come back clean. The check is working exactly as specified. It was never pointed at the world.

Groundedness inherits its framing from the attribution literature, where the AIS standard asks whether a generated statement about the external world is supported by the underlying source material it points to, validated through a two-stage human annotation pipeline (Rashkin et al., 2021, arXiv preprint). Read against that standard, groundedness is the wider question and faithfulness the narrower one, scoped to a specific retrieved set. That ordering is our reading of the two terms, not a claim the attribution work makes.

RAGAS computes faithfulness with no gold answers, which is the property that makes it usable before a gold set exists: reference-free evaluation across retrieval effectiveness, faithful use of retrieved passages, and generation quality, without ground-truth human annotations (Es et al., peer-reviewed, EACL 2024). Where that convenience runs out is covered in our read on what RAGAS certifies and what it leaves open.

Answer relevancy asks whether the response was on-question

Answer relevancy grades the fit between the question asked and the response produced. A relevant answer can be wrong. An irrelevant answer can contain a true statement. The metric was built to catch the response that wanders off the question, and it does that job and no other.

It gets confused with a neighboring measure. Answer coverage counts answers returned over requests sent, which is a question about whether a response exists at all. Relevancy is about the content of a response; coverage is about the denominator underneath every other rate on the page. Report relevancy alone and a pipeline that quietly drops one request in ten looks identical to one that answers everything.

Retrieval drift is the only metric here that needs a clock

The other five are point-in-time proportions. You run the eval, you get a number, you attach an interval. Retrieval drift exists only as a comparison between two measurements taken at different times, as the index changes underneath a fixed query set or the query distribution moves underneath a fixed index.

Drift can therefore only be reported if the earlier measurement was kept along with its confidence interval, scored on the same items, because a drift claim is a claim about a difference. Two point scores from two different question sets cannot support one. Where the same items were scored twice, a paired McNemar test is the instrument that separates a real move from run-to-run noise, and a pass-rate interval on each measurement tells you whether the gap was ever wide enough to argue about.

In practice a judge model computes every number on this board

Context precision and context recall can be computed from relevance labels the way any information-retrieval metric is, and retrieval drift is a difference between two measurements. In the frameworks this page names, all six arrive as a judgment rendered by a model against a rubric, which means each score carries that judge’s error rate on top of the pipeline’s, and drift carries it twice, once for each measurement it compares. Picture two judges, with round numbers standing in for measured ones: one agrees with human labellers on four cases in five, the other on nineteen in twenty. An identical faithfulness score from those two carries very different weight, and a dashboard renders the pair the same way.

The machinery for measuring judge reliability sits next door. The LLM-as-a-judge hub is the entry point, and Cohen’s kappa on judge-versus-human labels is the number to establish before any RAG metric computed by that judge is quoted to anyone outside the team.

Domain transfer thins that evidence further. A 2024 telecom-domain study re-implemented the RAGAS metrics and reported that the framework does not document how the numerical value of each metric is derived, and that only some of the metrics scored higher under correct retrieval in that domain (Roychowdhury et al., arXiv preprint). A metric validated on open-domain question answering has not thereby been validated on your contract corpus, and a single-domain preprint is thin evidence either way. Re-establish agreement on your own data.

The board says nothing about the requests that never came back

Every score here is conditioned on the subset of traffic that produced an answer. When membership of that subset correlates with difficulty, the conditioning distorts the number in a specific direction, which is the mechanism coverage conditioning names: a pipeline that declines the hard half of its traffic reports better metrics than one that attempts everything. Publish the share of requests the pipeline declined beside the metric board and that distortion becomes visible instead of flattering.

The generator’s identity travels the same blind spot. A fallback chain that re-issues a declined request to a second model changes which model wrote the answer you then scored, and none of the six metrics records that substitution. Faithfulness has no field for which model was faithful.

What holds all of this together is containment. A retrieval error becomes a wrong answer only when the generation step fails to stop it, so the fraction of retrieval faults held at that boundary is a containment rate in the same sense the rest of this site uses the term, and it is the one quantity on this page that grades the join rather than either side of it.

Each of these numbers has a page that owns the method behind it

A definition tells you what a metric means. It won’t tell you how many questions to run, where to put the gate, or which stage to blame when the score drops. Four other pages carry that load.

Evaluating a RAG pipeline beyond a single score owns the method: score retrieval and generation separately, put an interval on every metric, attribute each failure to the stage that produced it. Read it when you have the metrics and need the procedure around them.

When a score has already dropped and you want a shortlist of causes, the taxonomy is the faster route. RAG pipeline failure modes and the gate that stops each one works through five recurring modes, how each one propagates from retrieval into generation, the signal that detects it, and where its gate belongs.

Once a planner, a router and a self-correction loop sit on top of retrieval, the six metrics have to be measured per hop rather than once at the end, because every extra hop is both a recovery path and a new failure surface. Agentic RAG architecture and where each pattern breaks is written for that shape.

Where RAGAS wins and where it stops is the narrowest of the four, and it handles the named-tool question, including the statistics its point scores leave out.

Pick the stage you suspect, read the metric that grades it, then take the number to the method page rather than back to the dashboard. This page is written for someone who already has a board and wants to know what each number on it leaves out. If you are starting from a blank eval with no stage to suspect yet, the stage-split method is the first read, because it is the step that turns six scores into a diagnosis you can act on.