Answer relevancy

Answer relevancy is the degree to which a generated answer responds to the question that was actually asked, scored from the question and the answer alone, with nothing in the computation consulting whether the answer is correct. In a RAG evaluation it grades the generation step from the query side, penalizing three things at once: a reply that misses part of a multi-part question, one that wanders onto adjacent ground, and one padded with material the question never called for. It reads a different edge of the same generated text than faithfulness does, which scores that text against the passages the retriever supplied.

Averaging similarity scores across an eval set gives you a continuous mean rather than a proportion, and the interval that belongs on the number follows from that shape. A binomial interval of the kind you would put on a pass rate applies only once you threshold the score into pass and fail, and that threshold then becomes a parameter you have to publish with the result.

A second source of spread never reaches that interval. The score comes from a model call rather than a deterministic function, so re-scoring the same answers on a different day moves the number. That puts answer relevancy inside the same variance envelope as any other LLM-graded measurement, and it makes re-running the scorer on fixed inputs a part of the eval rather than a check performed on it.

In the pipeline it sits on the generation side of the retrieval-to-generation boundary, paired with context precision and context recall on the retrieval side, each of them answering one question and declining several others, which is why the RAG metrics are read as a set rather than singly. That split is what lets a RAG eval name the stage that broke instead of returning one blended figure. A relevancy mean falling across releases is one visible surface of retrieval quality degrading under a moved index, since answers drift off-question when the window stops carrying the right passage.

Read alone, a relevancy score certifies only that the model answered the question in front of it.

How to calculate answer relevancy

The standard construction scores the answer by reversing the direction of generation. Take the answer, prompt a model to write the question it would have been a good response to, repeat that several times, embed each regenerated question along with the original, and score the answer as the mean cosine similarity between them. That is the formulation published in the Ragas paper in the EACL 2024 system-demonstrations track, which drew its embeddings from OpenAI’s text-embedding-ada-002 and averaged similarity over n regenerated questions per answer (Es, James, Espinosa Anke and Schockaert, 2024). Cosine similarity runs from -1 to 1, so the score carries no mathematical guarantee of landing inside the unit interval, however reliably it does in practice.

Report the mean with an interval and the item count under it, the discipline an eval confidence interval asks of any eval statistic. Publish how many questions you regenerate per answer too, since that count sets how much run-to-run noise is averaged away before you see the score. No closed form fits the statistic cleanly, so resampling the scored items is the practical route to an interval; once you have thresholded into pass and fail, a Wilson or Clopper-Pearson interval fits what has become a proportion.

Instrument it at the generation boundary, on the answer as it leaves the model and before post-processing rewrites it, with the original question stored in the same record. Then decide what a refusal or an “I don’t know” reply scores. Dropping those rows lifts the mean by removing the cases the pipeline handled worst, so give them their own outcome bucket and carry the count.

Answer relevancy vs answer coverage

Answer coverage is answers returned over requests sent: the share of an intended sample that came back at all, once refusals, timeouts, truncations and rate limits have each taken their cut. It is a statement about the denominator under every other number in the report. Answer relevancy is a statement about the answers sitting inside that denominator.

The two names invite a wrong reading, because relevancy already penalizes an incomplete answer, which makes coverage sound like the completeness half of the pair. Answer coverage counts requests against answers, while answer relevancy scores the content of what those answers said.

Either figure can look healthy while the other is failing, and the two failures share no cause. A pipeline that declines the hard third of a question set posts a high relevancy mean on the two thirds it did answer, and that mean never mentions the third it walked away from, which is coverage conditioning operating on a RAG metric. A pipeline that answers everything can still post a poor relevancy mean at full coverage, and that combination points the investigation at generation rather than availability. Print the two figures adjacent, so a reader can see how much of the sample the relevancy mean rests on.

Answer relevancy vs faithfulness

Faithfulness is whether every claim in an answer can be inferred from the retrieved context, scored claim by claim against the passages the pipeline put in front of the model. Answer relevancy scores that same answer against the question instead. Where groundedness asks whether output traces to any source at all, faithfulness narrows the question to the retrieved context, and relevancy steps outside both to ask what the question wanted.

A fabricated answer aimed precisely at the question scores near the top of the relevancy range, because the questions regenerated from it look like the one that was asked, and nothing in the construction ever opens the context. Run the pair the other way and an answer that quotes a retrieved passage accurately and at length can score poorly, when that passage speaks to the neighboring question and the regenerated questions come back wrong for that reason.

The Ragas validation measured how differently the two behave as automatic metrics. On the WikiEval set, agreement with human annotators in pairwise comparisons came out at 0.95 for faithfulness and 0.78 for answer relevance, both reported as point accuracies with no interval attached. The same table’s GPT Score baseline reached 0.52 on answer relevance, so 0.78 improves on a weak floor. On that table, and on the one question set behind it, relevance was the harder of the two to automate. Whether it is also the noisier signal in your pipeline depends on your question mix and on the judge you run. Your own agreement numbers settle it for you.

Information retrieval judges a found document, relevancy judges a written answer

Information retrieval has used relevance for far longer, and means something narrower by it: the fit between a retrieved document and a query, assessed against a fixed test collection. That tradition is the one NIST convenes as the Text REtrieval Conference, across more than three decades of collections. The meaning is older, better established and correct in its own field. The object under judgment there is a document the system found; in RAG evaluation the object is an answer the model wrote, scored with no gold answer to compare it to.

The collision surfaces when someone asks for a relevance benchmark, meaning one number that ranks pipelines against each other. A relevancy mean is conditional on the question set that produced it. On one unchanged pipeline, a set weighted toward single-fact lookups scores higher than a set weighted toward multi-part questions, because the penalty for a partial answer can only bite where a question has parts to miss. A construction that survives scrutiny segments by question type, reports each segment’s mean with its interval and item count, and holds the question set fixed across every comparison. Built that way, the number can be compared against itself over time, which is the job a benchmark is doing.

Relevancy grades the generation half of a RAG pipeline alongside faithfulness, and neither number reads clearly until the retrieval scores sit on the page beneath them. The taxonomy of what breaks, and where the gate for each mode belongs, is worked through in RAG pipeline failure modes; the named-tool view, including where reference-free metrics stop carrying enough statistical weight, sits in our read on what RAGAS certifies.