Where RAGAS wins at RAG evaluation, and where it stops

For certifying a RAG pipeline, RAGAS is the obvious first pick. It scores faithfulness, answer relevance, and retrieval quality without asking you for a single gold answer, so you can wire it into a nightly job the same afternoon you hear about it. The real question is narrower than “is RAGAS good,” though. It is whether a RAGAS score actually settles the thing you need settled: is this pipeline safe to ship, and did this week’s change measurably improve it? For the first kind of question RAGAS earns its place. For the second, it returns a number that orders options but cannot size the gap between them.

The site behind this page points toward a reliability profiler, a pre-launch instrument whose design intent is to put statistical rigor around RAG scores; no number is measured yet, so the page quotes none of its own, and every RAGAS figure below resolves to the paper you can read yourself.

The decision scorecard

The asset on this page is one table. It maps each decision a RAG evaluation has to settle against what RAGAS actually delivers, the rigor gap it leaves open, and a verdict you can act on. RAGAS was published at EACL 2024 (System Demonstrations), so this is a peer-reviewed baseline, and the strengths below are real.

RAGAS reports its validation as three accuracy numbers, 0.95 for faithfulness, 0.78 for answer relevance, and 0.70 for context relevance, with no interval on any of them (Ragas, Table 1, EACL 2024, as of 2026-07). Those numbers are perfectly good as a headline. They cannot tell you whether the two-point score difference between your Tuesday pipeline and your Thursday pipeline is a real improvement or the same run sampled twice.

What RAGAS genuinely gets right

Start with the strength, because it is the reason RAGAS deserves the default slot. It is reference-free: you point it at questions, retrieved contexts, and answers, and it returns scores without a labeled gold set behind them (arXiv:2309.15217). Building a gold set for a retrieval corpus is slow and it goes stale every time the corpus changes, so removing that dependency is what makes fast iteration possible at all.

The second strength is the decomposition. RAGAS splits the pipeline into a retrieval question (does the context contain what the answer needs) and two generation questions (is the answer grounded in the context, and does it address the query). That split is real diagnostic value: a low faithfulness score with high context relevance points at the generator, and the reverse points at the retriever. Most teams instrument neither, so a framework that separates the two is a step up from a single end-to-end quality number.

What groundedness means in RAG evaluation

Groundedness is the degree to which every claim in a generated answer is supported by the retrieved context rather than by the model’s parametric memory or invention. A fully grounded answer asserts nothing the retrieved passages fail to entail, and an ungrounded sentence is a hallucination wearing a citation. RAGAS operationalizes groundedness as its faithfulness metric: a language model decomposes the answer into individual claims and checks each one against the retrieved context, and the score is the fraction the context supports. This is the dimension RAGAS validates most tightly, at 0.95 agreement with human annotators, so a faithfulness reading is the one RAGAS number you can lean on hardest. It grades generation, not retrieval: a perfectly grounded answer built on the wrong passages still scores high, which is why groundedness always has to be read next to a retrieval-side score.

The retrieval score you most need is the one you can least trust

Here is the specific limitation, in the authors’ own words. RAGAS validates unevenly across its three dimensions, and it is weakest exactly where a retrieval-augmented system usually breaks. Faithfulness tracks human judgment at 0.95 agreement; context relevance sits at 0.70. The paper is candid about why: “We found context relevance to be the hardest quality dimension to evaluate. In particular, we observed that ChatGPT often struggles with the task of selecting the sentences from the context that are crucial, especially for longer contexts” (Ragas, EACL 2024, as of 2026-07).

Retrieval is the half of a RAG pipeline that a generation-quality benchmark can never see, and it is the half RAGAS scores least reliably, on long contexts, which is precisely the regime production systems are moving toward. Whether a passage is “relevant” is also a construct-validity question before it is a scoring one: the metric grades a proxy for relevance, and on the long contexts where the proxy and the construct part ways, the authors report the model struggles to select the crucial sentences. Treat context relevance as a smoke alarm. It earns your attention when it fires, and a quiet reading still leaves the retriever unproven.

The scores arrive without an interval

The deeper gap is statistical, and it is not unique to RAGAS. RAGAS metrics are computed by a language model that extracts statements and checks them, which means the scores are LLM-as-judge scores, and they inherit the same bias that distorts any model-scored eval. A judge that prefers verbose or familiarly formatted answers tilts a faithfulness score the same way it tilts a preference ranking, and RAGAS ships no correction for it.

Compounding that, the scores arrive as bare point values. Because an LLM judge resamples its verdicts on each pass, re-running the same pipeline returns a spread of scores, and treating two points from that spread as a ranking is how a change that did nothing gets promoted. The fix is the standard one: report a confidence interval on every aggregated metric, repeat runs to expose run-to-run variance, and run a paired significance test before you call one pipeline version better than another. This is the discipline behind bias-corrected reporting with a calibration-aware interval, and it applies to a RAG metric as hard as it applies to a leaderboard. A number without an interval ranks; it does not certify.

Where propagation and attribution come in

RAGAS tells you a stage scored low. It does not tell you how far that failure travels. In a real pipeline a retrieval miss becomes an unfaithful answer, which becomes a wrong tool call in the agent that consumed the answer, and the score on each stage measured alone hides that chain. Tracing how a fault propagates across a pipeline, and attributing the final failure to the stage that caused it, is a different measurement than scoring three dimensions independently. It is the measurement that separates “context relevance is 0.70” from “this retrieval miss is why the answer shipped wrong.”

That is the seam a component-quality framework leaves open, and it is where propagation-aware reliability work fits alongside RAGAS. The two answer different questions, and a serious RAG pipeline usually needs both.

Context recall, context precision, and answer relevancy

The RAGAS library measures retrieval with two metrics the original paper folded into a single context-relevance score, and it scores the answer with a third. Here is what each one asks.

What is context precision? Context precision measures the signal-to-noise ratio of the retrieved context: of the chunks the retriever returned, are the relevant ones ranked at the top? A high score means the passages that actually matter sit above the filler. It can run reference-free, with the judge deciding which chunks are useful.

What is context recall? Context recall measures whether the retriever fetched everything the answer needs, or whether a required fact got left behind. RAGAS computes it by checking that each claim in a ground-truth answer is attributable to the retrieved context, so the standard context-recall metric needs a reference answer, which is the one place RAGAS’s reference-free property does not hold.

What is answer relevancy? Answer relevancy measures how well the generated answer addresses the question that was asked. RAGAS estimates it by generating candidate questions from the answer and comparing them to the original, which penalizes answers that are incomplete or padded with off-topic material. It is reference-free.

How do these relate to faithfulness? Faithfulness and answer relevancy both grade the generation step from different angles: faithfulness asks whether the answer is grounded in the context, answer relevancy asks whether it is on topic. Context precision and context recall grade the retrieval step. A pipeline can post a high faithfulness score while context recall quietly fails, because a well-grounded answer to an under-retrieved question is still wrong; when that happens the fault sits in retrieval, the way a RAG pipeline’s failure modes usually do.

Who should pick RAGAS

Pick RAGAS when you are iterating fast on a single RAG pipeline, you have no gold answers, and you need cheap directional signal on whether answers are grounded and on topic. For that job it is the strong baseline, and standing up a general-purpose eval framework first is over-engineering. It is a peer-reviewed, reference-free tool that decomposes the pipeline the way you actually debug it, and that is worth defaulting to.

Then hold its output to three rules:

1. Trust its context-relevance number as a hint, because its authors already told you that dimension is the shakiest.
2. Never let an interval-free score decide that one pipeline version beat another; wrap the metric in a confidence interval and a paired test first, the same standard you would demand of any reliability number worth acting on.
3. When you need to know why the answer was wrong, add the propagation-and-attribution layer RAGAS does not carry.

The research index and the reliability glossary are where those methods and their intervals live.