Rubric drift (LLM judges)

Rubric drift is the shift in an LLM judge’s applied standard across a run, a re-score, or a version change, while the rubric text sent to the judge stays byte-identical. A judge that scores an answer 4 one month and gives the same answer a 3 the next has drifted, and every comparison spanning that boundary is reading two instruments at once. It sits apart from the standing biases mapped on the judge bias pillar, such as preferring the longer answer or the one shown first. Those are stable offsets you characterize once and correct for, and drift moves the offset while you are using it.

Two things move a judge without anyone editing a rubric. The first is the model behind the alias. Our own pre-registered routing work, the study of what a premium tier quietly served when it rescued a refusal, found that fallback firing on 20 of 28 low-effort calls, with every rescued answer coming from the cheaper model. The name in the request and the model that answered came apart on most calls.

Editing the scoring prompt does it too, even when the edit reads as a harmless clarification. A June 2026 arXiv preprint on attributing drift to the judge or the system measured a stricter rewrite of exactly that kind, reserving the top score for flawless work and taking the lower score when torn. It pulled mean scores down on four of the five HelpSteer2 rubrics, by 0.043 to 0.152 points, with verbosity the exception at 0.025 up. A silent swap of the judge model pushed four of the same five the other way, by 0.022 to 0.072, with correctness the exception at 0.008 down.

Because rubric drift is a change in the measuring instrument between two measurements, the scores of the thing being measured cannot detect it on their own. A monitored score that falls is ambiguous over a worse product and a stricter judge, and both readings predict the same downward line.

Resolving it needs something whose true value cannot have moved, which is the reasoning behind an anchor set. That is a fixed pool of human-labeled items the live judge re-scores on a steady interleave, so the judge-versus-human gap becomes a series in its own right. Rubric drift is why a judge’s confidence-to-accuracy mapping has a shelf life, and it is the specific way eval reproducibility fails when code, seeds and data are all pinned and the numbers still move. It is the entry in the judge-bias glossary set that neither a swap test nor a calibration set can see, because both of those hold the rubric fixed. Publish the resolved judge identifier and the rubric version beside every score, the way you would publish an interval.

How to measure rubric drift

Freeze a set of items with human labels, keep them out of the pool the product is optimized against, and have the live judge re-score that set on the cadence of production traffic. Track the judge-versus-human gap rather than the raw production score, since the anchor’s true value is fixed by construction and any movement in the gap belongs to the judge. Report that movement as a change carrying its own interval rather than as two point estimates set side by side. Both judge versions scored the same items, so the comparison is paired and McNemar on the disagreeing pairs is the test that fits. Where the anchor labels are categorical, watch chance-corrected agreement with the humans across versions, since raw exact-match agreement overstates how far a judge and a human really agree.

In that preprint’s experiments the design caught a silent version bump as judge drift in 60 of 60 runs with no judge-to-system misattribution, and attributed a stricter-prompt change correctly on 110 of 120. The rolling z-test it was measured against false-alarmed on 75% of drift-free streams (single-author preprint, not yet replicated). Instrument it at the judge call, logging the model identifier the provider resolves to alongside a hash of the rubric text.

Rubric drift vs judge inconsistency

Judge inconsistency is disagreement between two scorings of the same item by the same judge, same rubric, same version, minutes apart: the test-retest axis a self-consistency check measures. Norman, Rivera and Hughes, in a June 2026 arXiv preprint covering 21 judges from nine providers over roughly 541,000 individual judgments, report test-retest reliability above 0.95 for two of the production-deployed judges they tested, so this axis can be the healthy one. Rubric drift is a movement in the center of that same distribution between two points in time.

A green consistency check licenses nothing about drift. A judge can be near-perfectly self-consistent and still drift, scoring tightly around one standard this month and tightly around a lower one after a version bump, so the self-check passes while the time series underneath it has broken. A judge can also be noisy without drifting, holding its mean while the spread stays wide. Consistency bounds the weight one score can carry, and a drift check decides whether last quarter’s number belongs on the same chart as this one.

Rubric drift vs criteria drift

Criteria drift is the name Shankar and colleagues gave, in their UIST 2024 study of aligning LLM-assisted evaluation with human preferences, to something the humans do. Graders need criteria in order to grade outputs, and grading outputs is how they discover what the criteria are, so the rubric keeps changing as the evaluation proceeds. Their finding is that some criteria depend on the outputs observed rather than being definable in advance, which unsettles any workflow that assumes a fixed rubric.

Rubric drift is what shows up in the scores when nobody has edited anything at all.

The two interact badly, and in a predictable direction. A criteria-drift edit is legitimate and usually an improvement, and it still invalidates every score produced under the previous wording, so a healthy revision registers downstream as a discontinuity. Version the rubric the way you version code, record which labels came from which version, and re-baseline the anchor set after any edit that moves the scale rather than the phrasing.

A judge is an agent too, so one system can drift both ways

These two share a word and very little else. Role drift is an actor problem: an agent gradually abandons the role it was assigned over a long run, answering as a generalist when it was scoped to one tool. Rubric drift is an instrument problem, where the thing doing the scoring changes scale underneath a fixed rubric. A judge is itself an agent, so one system can carry both, and they separate by where the change appears: role drift in what the agent does, rubric drift in a score series whose subject matter never changed.

Rater drift in educational assessment

Educational measurement has worked on this for decades under the name rater drift: a human scorer’s severity moving over a scoring session, so an essay marked in the first hour and an equivalent one marked in the sixth are not treated alike. The evidence there is more careful than the borrowed phrase suggests. Leckie and Baird, analyzing England’s 2008 national curriculum English writing test in the Journal of Educational Measurement (2011), found no significant systematic drift in rater severity, and did find that rater severity was significantly unstable over time. Those are two separate failures, and the distinction transfers intact.

A judge whose standard trends in one direction biases every comparison in that direction; a judge whose standard wanders without trending leaves the mean alone and widens the real uncertainty around every score, which a single-run interval understates. Test for each: a trend test on the anchor gap catches the first, the spread of that gap across re-scores catches the second.

Judge calibration tells you where the judge’s standard sits, and rubric drift tells you whether it stays there. The gate order for the standing biases is in the checklist that gates a ranking, and the fuller case for characterizing a judge before publishing anything it measured lives in our guide to using an LLM as a judge. If you cannot name the judge build and the rubric hash behind a score, then that score cannot be compared with next quarter’s.