Retrieval drift

Retrieval drift is the decay of a retrieval stage’s quality across calendar time in a system that is otherwise running normally, scored as the difference between what a fixed query set returns today and what the same queries returned at an earlier baseline. It reads across two moments, which is where it parts company with context precision and context recall. Each of those scores one snapshot and holds no information about direction.

Because the quantity is a difference between two measurements, it inherits the uncertainty of both, and a delta published without an interval cannot say whether retrieval moved or the sample did. Score the same frozen, labeled query set at both ends so the comparison is paired, which strips out the query-to-query variance that would otherwise swamp a small delta. Draw a fresh sample each cycle instead and the figure moves with the sample, hiding whatever retrieval did.

In the lane’s metric graph it sits behind the retrieval-side scores rather than beside them. Precision and recall give the level, drift gives the movement, and both readings come off the same labeled set. Downstream, faithfulness and answer relevancy hold their numbers while retrieval ages underneath them, since each scores the generator against whatever window it was handed. That is the slow entry in the failure taxonomy you monitor a RAG pipeline against, and no single-snapshot score reaches it. Re-score on a schedule, or the eval describes only the day you happened to run it.

How to measure retrieval drift

Freeze a query set with relevance labels, record what the retriever returns for each query, then re-run that same set on a cadence. At a chosen k the per-query outcome is binary, whether the answer-bearing chunk arrived, so two runs form a paired 2×2 table: held, gained, lost, still missing. The McNemar test for paired eval runs reads that table directly and returns an exact p-value with an interval on the difference. The paired design in the sample size and power calculator says how many labeled queries a fall of the size you care about needs, and a set too small to see a five-point drop reports reassuring flatness for a year.

Size the set before the first run rather than after the first surprise.

Instrument at the retriever’s return, upstream of reranking and upstream of the prompt builder. A token budget drops chunks between those points, so a fall measured at the assembled window can’t separate a retriever that stopped finding the passage from an assembly step that stopped fitting it in. A calculator for chunk size against the context budget covers what a given budget admits.

Labels age with the corpus, which is the awkward part of the procedure. A label set written against a corpus that has since doubled carries judgments that never saw the documents now competing for the top k, so a relevant chunk counts as irrelevant by omission and pulls the score down with no regression behind it. Re-label a sample of each cycle’s new top k.

The index side and the query side move separately

Two independent things travel under one drift number. On the index side documents arrive and get superseded, the chunking scheme changes, the embedding model is swapped, or the approximate nearest-neighbor parameters are retuned. An embedding upgrade is the sharpest of those, since vectors written by the old model and the new one do not sit in a comparable space, so the corpus has to be re-encoded and the index rebuilt before any score means anything. On the query side nothing inside the system changes: users bring different questions than they used to, and a retriever tuned on one query distribution can lose ground on another.

That second case is easy to under-rate, because the in-domain number gives no warning of it. The BEIR benchmark evaluated ten retrieval systems across eighteen datasets and reported that “in-domain performance is not a good indicator for out-of-distribution generalization” (Thakur et al., NeurIPS 2021 Datasets and Benchmarks track, peer-reviewed). BM25 trailed neural approaches by 7 to 18 points in domain on MS MARCO and still held up as one of the stronger baselines outside it. A score earned on the queries you already had is weak evidence about the queries you are about to get. The two mechanisms take different fixes, so a report that leaves them fused sends a team off to re-embed a corpus when the query mix is what moved.

Retrieval drift vs eval reproducibility

Eval reproducibility is whether re-running an eval on the same data with the same parameters lands on the same result, in the computational sense of the word, where nothing about the world is allowed to change between the two runs. Retrieval drift is the reading you take when one thing is deliberately allowed to change, and that thing is time.

A pipeline can be perfectly reproducible and drifting hard, when every run is deterministic given the index and the index is a month older than the baseline. The reverse case is irreproducible with no drift at all: approximate nearest-neighbor search returns a different neighbor set as shard layout, graph entry points and concurrency shift under it, so two runs an hour apart disagree while the long-run level stays flat. Measure that run-to-run spread first, since it sets the floor beneath any drift you can claim. A delta sitting inside it is a reading of the retriever’s own noise, which is what an eval confidence interval exists to make visible.

Retrieval drift vs rubric drift

Rubric drift is a different instrument coming loose: an LLM judge’s effective standard moves while the rubric text it receives stays fixed, so two scores produced under the same rubric stop sitting on the same scale. Retrieval drift moves the evidence, rubric drift moves the ruler, and on a dashboard the two arrive as the same falling line.

They separate cleanly only when the retrieval score avoids passing through a model judge. Labels applied by hand, or frozen from an earlier human pass, cannot shift when the judge model is upgraded, so a delta computed against them is attributable to retrieval. Hand relevance labeling to a model and the two failures superimpose, since a stricter judge and a degraded retriever both push context precision down. A third term shares the word and belongs to neither cluster: role drift is an agent’s remit sliding across a long run, a multi-agent failure rather than a retrieval or a judging one.

Drift already means something in machine-learning monitoring

The monitoring literature reached the word first, and its definition is narrower than this one. Gama and colleagues define concept drift as “an online supervised learning scenario when the relation between the input data and the target variable changes over time” (Gama et al., ACM Computing Surveys 46(4), 2014, peer-reviewed). Their survey splits it into real concept drift, where the conditional distribution p(y|X) changes, and virtual drift, where the incoming distribution p(X) changes without affecting p(y|X). A moving query mix is the p(X) case, and a corpus whose correct passage for an unchanged query has been superseded is the p(y|X) case, which is what a stale index produces.

Monitoring products carry that heritage into a single drift score over embedding or feature distributions, and a score of that shape answers a narrower question than a retrieval eval asks. It reads the inputs and nothing that came back. It never scores a retrieval against a relevance label, so it can’t tell an index that changed and still returns the right passage from one that changed and stopped. Keep the distribution monitor as an alarm saying when to re-score, and keep the labeled frozen query set under the paired test as the score itself.

A RAG report that publishes a level with no delta beside it has answered half of what was asked. The method for scoring retrieval and generation separately, so a fall is attributable to the stage that produced it, is worked through in our RAG pipeline evaluation method, and the wider map of RAG metrics sets out the single question each neighboring score can answer.