Faithfulness (RAG evals)

Faithfulness in a RAG evaluation is the share of the claims in a generated answer that can be inferred from the context the retriever supplied for that question. It scores one step, generation, against one reference: the passages that actually reached the model. That puts it downstream of how much of the retrieved set deserved to be there and of how much of the relevant material was retrieved at all.

The quantity is a ratio of two counts, and both of them need publishing. Es and co-authors, presenting the Ragas framework at EACL 2024, define faithfulness as the statements in an answer that the context supports divided by the statements extracted from that answer, which makes the figure a proportion with every obligation a proportion carries. An answer decomposed into three statements and one decomposed into twelve both produce 0.67 with very different evidence underneath, and a suite average pooling claims from long answers with claims from short ones is a weighted mean nobody declared.

Inside a pipeline, faithfulness sits at the retrieval-to-generation boundary, and a score there means only as much as the retrieval scores beside it on the RAG evaluation metric board. A generator that stays faithful while the index quietly ages holds its number while the product degrades, which is what an index and a query distribution drifting apart does to a pipeline. One rule covers the reporting: quote faithfulness with its claim denominator and an interval, or it is an assertion about an answer set nobody can size. Before any of that, the term has to be pulled apart from groundedness, which it gets swapped for constantly.

Faithfulness vs groundedness

Groundedness asks whether an output traces to an identified source at all: some passage, document or tool return a reader could open and check, whether or not a retrieval step produced it. Faithfulness asks the narrower question of whether the claims follow from the specific context this request retrieved.

A high faithfulness score does not imply a well-grounded answer, and a well-grounded answer does not imply a scorable faithfulness result. An answer can score a perfect 1.0 on faithfulness while resting on nothing: every claim follows cleanly from a retrieved chunk that happens to be a stale internal note or a pasted user message that found its way into the index. Faithfulness checks entailment from the window and never asks whether the window carries authority. Running the other way, an answer produced with no retrieval at all can be well grounded, when the model names a checkable source; faithfulness has no reference to score against there and is simply undefined.

Faithfulness moves when the generator changes: a new model version, a rewritten prompt, a longer window. Groundedness moves when the corpus underneath it changes, and that is usually a different team on a different release cycle. Both can regress in the same week while only one of the two numbers notices.

Publishing one of the two figures leaves the other component unscored.

How to calculate faithfulness

Decompose the answer into standalone claims, each one assertion that can stand or fall on its own, then judge every claim against the context. Faithfulness is the supported count over the extracted count. Ragas runs both steps with a language model, prompting once for extraction and again for verification. That is what makes the metric reference-free, and it is also what puts a second model inside your measurement, so what a reference-free score can and cannot certify is worth settling first.

Score against the context string the model actually received, after truncation, rather than the ranked list the retriever returned. When a passage is dropped between retrieval and the prompt, an evaluation reading the retriever’s output records a generation failure that never happened. In an agentic pipeline the reference is every passage shown across the run’s retrieval hops, since each extra hop is a recovery path and a new failure surface at once and a model can ground a claim in a chunk it saw four steps earlier.

Report the result with an interval, and choose the unit of resampling deliberately. Claims inside one answer are correlated: one question, one context window and one generation produced all of them, so an interval computed as though every claim were its own trial comes out too narrow. Score each answer, then put the interval on the answers, with n as the number of questions instead of the number of claims. The reasoning behind an eval confidence interval carries over unchanged, and a Wilson or Clopper-Pearson interval covers each answer collapsed to a pass or a fail.

Check the verifier itself before any of that means anything: score a sample of claims against human labels and publish the agreement, which Cohen’s kappa on judge-versus-human labels gives you directly. Correct the raw rate for the verifier’s sensitivity and specificity once you know them, which is what a bias-correction calculator does. Say which answers were scorable as well: refusals, empty generations and answers yielding no extractable claim all leave the denominator, and a pipeline that declines the hard questions scores better on the ones remaining. That is a rate quietly conditioned on who answered, so publish answers returned over requests sent beside the score.

Faithfulness vs answer correctness

Answer correctness asks whether the answer is true. Faithfulness asks whether it follows from what was retrieved, and the gap between those questions is old enough to have its own literature. Maynez and co-authors drew the line for abstractive summarization at ACL 2020: a summary counts as faithful when it says only what the source supports, and factual when it holds up against world knowledge. Content absent from the source document that happens to be factually correct anyway is a factual hallucination, and their annotators found it in real summaries.

Both directions turn up in ordinary production traffic. Retrieve a superseded policy page and the generator answers faithfully and wrongly, a 1.0 documenting a retrieval problem. Let a model fill a gap from parametric memory with something true and faithfulness marks the claim unsupported, which reads correctly for a grounding metric and misleadingly for a correctness one. So a faithfulness score certifies nothing on its own about whether the answer is right, and a drop in it is worth tracing back to the stage that produced it before anyone rewrites the prompt. Scoring retrieval and generation separately is the whole method behind evaluating a RAG pipeline past one blended number.

Faithfulness in chain-of-thought and interpretability

The same word carries a different technical meaning one field over, and the two turn up in the same conversation often enough to be worth separating. In interpretability research, a faithful explanation is one that reflects the process a model actually used to reach its answer, so the object under test is the model’s own reasoning rather than a retrieved document. Lanham and co-authors measured that directly in “Measuring Faithfulness in Chain-of-Thought Reasoning” (arXiv preprint, 2023), inserting mistakes into a model’s stated reasoning and paraphrasing it to see how much the final answer depended on the steps shown. They reported that larger models produced less faithful reasoning on most of the tasks they studied.

A high RAG faithfulness score says nothing about that property. A model can restate its retrieved context accurately while the chain of thought printed above the answer is decoration. Keep the two under separate names in any report, faithfulness to context for the RAG metric and reasoning faithfulness for the interpretability property, because a reader who merges them will take a grounding result as evidence about a model’s transparency.

Faithfulness scores the generator against the passages it was shown, and whether the answer was on the question at all is the other generation-side reading, which a pipeline can fail while passing this one. The fuller account of how a retrieval error becomes a wrong answer, and which gate stops it at which boundary, lives in the RAG failure taxonomy. A team scoring one-sentence answers won’t get much out of the ratio, since a denominator of one or two claims turns it into a coin flip with an interval wide enough to say so.