Context precision (RAG evals)

Context precision is a retrieval-side metric in RAG evaluation: of the chunks a retriever returned for a question, the fraction that were relevant to answering it. The denominator is what came back, so a high score says nothing about the answer-bearing passage still sitting unretrieved in the corpus. Context recall covers that half of the retrieval step, and the two are read as a pair.

It sits on the retrieval side of the stage split that scoring a RAG pipeline properly turns on, next to context recall, while faithfulness and answer relevancy grade what the generator did with the window. A score that sags month over month with no code change behind it is retrieval drift; a low one is the retrieval-side signature of answers drifting off-topic, cataloged in the RAG failure taxonomy. Publish the k alongside the score, always; the same pipeline reports a different number at every k.

How to calculate context precision

Each query’s context precision is a proportion over a denominator you chose, and k stays small because the window has to hold the chunks, so the per-question score lands on a coarse grid: at k=5, a score of 0.6 means 3 of 5 and nothing finer. The number worth publishing is the mean across a query set, which brings a second denominator, the count of questions evaluated. Both belong in the report: two teams quoting 0.72, one over 40 questions at k=5 and one over 400 at k=20, aren’t reporting the same quantity. Bracket the mean with an interval, and pick the form from how you scored, because a mean of fractional per-question scores is a different object from a pass count.

Run the retriever on a question, label each returned chunk relevant or not, then divide the relevant count by the number of chunks returned, and take the mean of the per-question scores across the eval set. Two routes lead to that interval. Threshold each question to a pass or a fail against a cut-off you publish, and the result is a proportion the pass-rate interval calculator covers, with the question count as the denominator, since chunks drawn from one retrieval event are not independent trials. Keep the fractions, and no closed form fits, so a bootstrap over the question set is what supplies the width.

Labeling is the expensive half of the procedure. Human relevance judgments are the reference standard and they scale badly, so most teams hand the judgment to a model, at which point the retrieval metric inherits whatever the judge does wrong. When a model assigns the labels, its calibration against human judgment is part of your retrieval number, and the biases that can flip a ranking apply to grading relevance the same way they apply to grading answers.

Instrument it at the point where the prompt is assembled. Reranking, deduplication and a hard token budget all sit between the retriever’s return and the final window, so a chunk the prompt builder dropped was never in the context the generator read. The chunk size and context budget calculator shows what a given chunk size and k consume, which is usually what fixes k before any precision number does.

The procedure above leaves out order, and order is where two different metrics have grown under one name.

Two different quantities ship under the name context precision

The plain form ignores rank: three relevant chunks out of five score 0.6 whether they arrive first or last. Ragas, the RAG-specialist metric library, documents its Context Precision@K instead as the sum of precision@k at each rank, weighted by whether the chunk at that rank is relevant, divided by the number of relevant items in the top K. That construction is average precision computed over the retrieved set, and it drops sharply when an irrelevant chunk is promoted to the first position, where the plain form does not move at all.

Both are defensible, and order matters to a generator because a model does not read a long window evenly: accuracy is highest when the relevant passage sits near the beginning or the end of the input, and it degrades when the passage sits in the middle (Liu et al., TACL 2024, peer-reviewed). The rank-aware form is the closer proxy for what the generator will do with the window, and the plain form is the cleaner description of how much of the window is noise. State which construction produced your number, next to the number. A score lifted from another team’s dashboard becomes comparable to yours only once the constructions match, one of the things a metric-by-metric map of RAG evaluation has to settle.

Context precision vs context recall

Context recall is the share of the information an answer needs that actually reached the retrieved set: its denominator is what the question required, and its numerator is how much of that arrived. Context precision inverts the denominator and scores what arrived.

One mechanism pulls them apart when k is tight, and a different one pulls them apart when k is wide. An aggressive reranker with a tight top-k returns three clean, on-topic chunks and leaves behind the fourth, the one carrying the figure the question asked about: precision near the ceiling, recall short, and an answer that is well grounded and incomplete. Widen k and the answer-bearing passage almost certainly arrives, escorted by a dozen that merely look similar, which reads as recall high, precision low, and a generator picking the right passage out of a crowd. The arithmetic makes the trade mechanical: a non-relevant chunk added at rank k+1 leaves the recall numerator untouched and raises the precision denominator, so precision falls as k grows whether or not retrieval improved.

Score both at the same k, in the same run. A change that lifts one and drops the other has moved the operating point along that trade, and only the pair makes that visible. End-to-end answer accuracy on its own cannot say which half of the pipeline earned a win, which is the gap that opens up in the agentic-search versus vector-retrieval question.

Context precision vs precision in information retrieval

Precision in information retrieval is the fraction of retrieved documents that are relevant, a definition that predates retrieval-augmented generation by decades (Manning, Raghavan and Schütze, Introduction to Information Retrieval, Cambridge University Press, 2008). Context precision is that arithmetic pointed at a context window, and the rank-aware variant above is the same book’s average precision restricted to the retrieved set. The formulas carry over without change, but what reads the retrieved set is different, and that difference decides what a low score costs.

A person scanning a result page skips an irrelevant hit at rank two and loses nothing else. A generator reads all k chunks in one pass and has no equivalent skip, so a distractor competes with the answer-bearing passage inside the same window and spends tokens the budget has to cover. The search-engine habit of raising k until recall saturates therefore transfers badly, since every extra chunk is another passage the model reads.

A second collision sits with the measurement sense of the word. In metrology and in statistics, precision describes repeatability, the spread of repeated readings of the same quantity, which is close to the reverse of what this metric reports. A retrieval stage can score high on context precision and still be imprecise in that older sense, because nothing in the score says it won’t return a different chunk set on the next run against the same index. That raises the separate question of whether a re-run lands in the same place, which no relevance label answers.

Two retrieval-side numbers make a retrieval stage scoreable on its own, ahead of any answer-side metric, and precision is the half that reports what the retriever returned rather than what it missed. Scoring retrieval and generation separately, then attributing each failure to the stage that produced it, is set out in the pipeline evaluation walkthrough, and our assessment of RAGAS covers why a reference-free score is a starting point rather than a verdict.