RAG Retrieval Quality Calculator: Precision@K, Recall@K, nDCG, MRR, MAP

Each query is a ranked list of retrieved ids and a set of relevant ids, scored at a cutoff K. With hits@K the number of relevant ids among the first K positions and R the number of relevant ids for the query: Precision@K is hits@K / K; Recall@K is hits@K / R; Hit@K is 1 when hits@K is at least 1; reciprocal rank (RR@K) is 1 / rank of the first relevant id if that rank is within K, else 0, and mean reciprocal rank (MRR) is its mean over queries; average precision (AP@K) is the sum of Precision@i over the relevant ranks i within K, divided by R, and mean average precision (MAP) is its mean; DCG@K sums each rank's gain divided by log2(rank + 1), IDCG@K is the DCG of the relevant ids in their best possible order, and nDCG@K is their ratio. Every score sits in 0 to 1. The suite figures are macro-averages: each scored query weighs the same. The definitions and the judged-collection setting are those of Manning, Raghavan and Schütze (2008), chapter 8; the gain-and-discount construction is Järvelin and Kekäläinen (2002). What each of these metrics certifies, and what it stays silent about, is worked through in the RAG evaluation metrics reference.

Conventions, and the variants they replace. Several of these metrics have more than one accepted form, so the tool states its choice beside each number and lets you switch where the choice is a live one. Precision@K divides by K even when fewer than K ids came back, so an empty slot costs what a wrong id would (the trec_eval P_K rule). Recall@K and AP@K divide by every relevant id by default (trec_eval recall_K and map_cut_K), which means a query with more relevant ids than K can never score 1; the "min(relevant, K)" option divides by the most that could fit in K instead, the R-precision-style form. RR is read at K, so a first relevant id below K scores 0; set K at or above the longest list to read the full-list MRR of trec_eval's recip_rank. DCG uses the linear gain (the grade itself, as trec_eval and scikit-learn do) by default; the 2^rel − 1 gain of Burges et al. (2005) is a switch and changes nothing for 0/1 labels. IDCG ranks every relevant id you list, retrieved or not, so a relevant id the retriever never returned still lowers nDCG. Ties among graded ids do not move IDCG; the retrieved order is yours, so it has no ties.

Two edge rules. A query whose relevant field is empty cannot be scored: Recall, AP, and nDCG are 0 / 0 and the other three would say nothing about the retriever. Such queries are excluded from every average and counted in the summary; if you meant "the corpus holds no answer for this query", that count is the figure to report. A repeated id in a retrieved list keeps its rank slot and only its first occurrence can score; the repeat counts as a non-relevant slot, so a retriever that returns the same chunk twice in its top K pays for it in Precision@K exactly as it would for a wrong chunk. Repeats are counted per query and in the summary.

Worked example for nDCG, small enough to check by hand. Six retrieved ids with grades 3, 2, 3, 0, 1, 2 in rank order. Linear gain: DCG@6 = 3 / 1 + 2 / 1.585 + 3 / 2 + 0 + 1 / 2.585 + 2 / 2.807 = 6.861. The ideal order of the five relevant grades is 3, 3, 2, 2, 1 (the judged non-relevant id contributes nothing): IDCG@6 = 3 + 3 / 1.585 + 2 / 2 + 2 / 2.322 + 1 / 2.585 = 7.141, so nDCG@6 = 0.961. If two more relevant ids with grades 3 and 2 exist but were never retrieved, IDCG@6 becomes the DCG of 3, 3, 3, 2, 2, 2 = 8.740 and nDCG@6 falls to 0.785 on the same retrieved list. For MAP: a query with relevant ids at ranks 1, 3, 6, 9, 10 out of five relevant has AP@10 = (1 + 2 / 3 + 3 / 6 + 4 / 9 + 5 / 10) / 5 = 0.622; a second with relevant ids at ranks 2, 5, 7 out of three has AP@10 = (1 / 2 + 2 / 5 + 3 / 7) / 3 = 0.443; MAP@10 = 0.533. All four figures are among the unit tests.

The preloaded sample. Ten queries at K = 5. Nine are scored; q08 lists no relevant ids and is excluded. Recall@5 averages 0.676. Five queries are full, with every relevant id inside the top 5 (q07 finds both of its ids at ranks 1 and 2, AP 1; q02 finds its single id at rank 3, AP 0.333). Two are partial: q04 finds 3 of 4, and q09 finds 2 of 6, which is a recall of 0.333 by every relevant id or 0.400 by min(relevant, K). One is a complete miss (q06, whose relevant ids never appear) and one is a late hit (q03, first relevant id at rank 6, so K = 6 would reach it; in the sweep, Hit@10 is 0.889). q10 repeats d05 at ranks 1 and 2; the repeat scores as a non-relevant slot, so its Precision@5 is 0.4 rather than 0.6. Hit@5 is 7 of 9 = 0.778 with a Wilson interval of 0.453 to 0.937: nine queries certify very little, which is the point of showing the band.

Averages and their intervals. Hit@K is a 0/1 outcome per query, so its mean is a proportion of queries and carries the shelf's Wilson score interval (Wilson, 1927), the same construction the pass-rate confidence interval calculator reports; the query is the trial, because chunks from one retrieval event are not independent draws. The other means average fractional per-query scores, which are not Bernoulli draws, so their bands come from a percentile bootstrap that resamples the query set 1,000 times with a fixed seed (Efron, 1979): the interval reflects how much the average would move under a different draw of queries like these, and a rerun reproduces it exactly. Both bands widen fast below a few dozen queries. Two named cousins are reported for orientation. The RAGAS context precision@K, defined in the context precision glossary entry, is average precision normalized over the relevant ids that were actually returned in the top K rather than over R, so it equals AP@K only when every relevant id sits within K; it is shown as its own row. Recall@K here divides by ids an assessor labeled relevant to the query, which is a different denominator from RAGAS context recall (evidence one reference answer needed), a distinction the context recall glossary entry works through. Where the judge-scored context metrics these ranked-list figures sit beside earn their keep, and where they run out, is assessed in our read on RAGAS as a rigorous RAG evaluation.

Honest limits. Everything here trusts the relevance labels: an incomplete judged pool undercounts recall and can misplace IDCG, and labels produced by a model judge carry that judge's error rate into the score. Every figure is tied to its K, its label set, its query set, and the day it was run; re-scoring one frozen query set later is how retrieval drift is read, and a single run cannot show it. Under the default denominator recall never falls as K grows while precision tends to, so a gain quoted without its K is not evidence, and comparisons hold only across the same queries at the same K. The averages weigh every query equally, so a suite dominated by single-relevant-id queries reads differently from one with many relevant ids per query. A multi-hop or agentic retriever has no single top-K; score the union of what it pulled and say so. The measurement discipline that keeps retrieval scores apart from generation scores is set out in how to evaluate a RAG pipeline.

Formula: nDCG@K = DCG@K / IDCG@K, DCG@K = Σ gain(rel_i) / log2(i + 1) over ranks i = 1..K, gain = rel or 2^rel − 1

See Sources below