Judge calibration (LLM evals)

Judge calibration is whether an LLM judge’s stated confidence tracks its accuracy: across the verdicts issued at a given confidence level, how close the share that held up sits to the confidence claimed. Say a judge reports 90% confidence on 100 pairwise verdicts and 71 of them survive human review; that band is overconfident by 19pp, whatever the headline accuracy looks like. That is a separate reading from how often the judge and a human labeler land on the same verdict, which scores the labels and discards the confidence sitting next to them.

Calibration is a property of a distribution of verdicts rather than of any single verdict, so it can only be estimated band by band, and every band is a proportion resting on however many verdicts fell into it. A band holding eleven verdicts carries an interval wide enough to swallow the gap you are trying to detect. Expected calibration error, the figure most teams quote, therefore moves with the number of bins chosen and with the item mix the judge saw. Publish the per-band counts and the item distribution beside it and another team can check the number. Publish it alone and it’s a house metric.

Judges do not arrive calibrated, and a fresh one is best assumed overconfident until measured on your own items. A 2025 preprint on overconfidence in LLM-as-a-judge setups reports that predicted confidence significantly overstates actual correctness across the judges its authors tested. A 2026 preprint that put four models through 24,000 trials of answering benchmark questions with a self-reported confidence found expected calibration error spanning 0.122 for Claude Haiku 4.5, at 75.4% accuracy, up to 0.726 for Kimi K2, at 23.3%. Neither is peer-reviewed, the second measured self-assessment rather than judging another model’s output, and neither puts an interval on those errors, so read the spread rather than the decimals.

Calibration starts doing real work the moment a confidence score decides where a verdict goes. Sending only low-confidence verdicts to human review saves annotation budget when low confidence predicts error, and spends it on easy items when it does not. Our hub on testing the evaluator itself splits a judge into agreement, calibration and bias, and the wider map of how an eval number lies names calibration drift as one of four mechanisms. Calibration has a row in our map of what each judge term actually names, where it stands for the scale offset that shifts every verdict at once rather than for one of the biases, and every row there carries the test that measures it.

A confidence score earns its place in an escalation rule only after the calibration behind it has been measured.

How to calculate judge calibration

Score a held-out set of items whose human labels the judge never saw, and require a confidence with every verdict. Sort the verdicts into confidence bands, then for each band compute the share that matched the human label, which gives that band’s empirical accuracy against the confidence it claimed. Average the gap across bands, weighting each by the verdicts it holds, and you have expected calibration error. The largest single-band gap is the maximum calibration error, and it is what a threshold actually meets, since a threshold sits inside one band rather than across the average.

Each band’s accuracy is a proportion, so it carries an interval, and the Wilson and Clopper-Pearson calculator bounds a band the way it bounds a suite. Report bands holding fewer than a few dozen verdicts with their counts visible, since folding them into a headline is where a bin scheme starts flattering the judge. Putting an interval on the aggregate error means resampling the scored verdict set, on the same reasoning as any eval confidence interval.

Instrument it where the judge writes its verdict, which in most pipelines is the row already holding the score, the rationale and the item id. Add the confidence field, keep the human label alongside for the subset you annotated, and the calibration table falls out of a group-by. One fix from the classifier literature will not transfer: temperature scaling, which Guo and colleagues showed at ICML 2017 to be an effective post-hoc correction for an overconfident network, needs the logits that a judge behind a vendor API rarely returns.

Judge calibration vs judge agreement

Agreement is the rate at which two raters assign the same label to the same item, corrected for the agreement chance alone would produce, which Cohen’s kappa reports and the inter-rater reliability calculator computes on a judge-versus-human pass/fail set. Calibration scores the thing an agreement statistic throws away, the confidence attached to each of those labels.

Agreement and calibration can each read well while the other reads badly. A judge can match human labels on 82 of 100 items, a raw match rate nobody has chance-corrected yet, while reporting 95% confidence on every one: agreement looks respectable, calibration is broken, and confidence-based routing misfires. A judge can also be well calibrated and agree poorly, reporting 60% confidence on a task where it is right about 60% of the time, which is an honest instrument too weak for the job. A judge report that omits either figure leaves an escalation rule resting on an assumption nobody tested.

Judge calibration vs judge bias

Bias is a systematic, directional shift in a judge’s verdicts caused by a feature of the response that should carry no weight. Documented forms include verbosity bias, where length stands in for quality, a judge favoring its own model family, and which candidate was shown first, a confound that only appears once you adopt pairwise rather than pointwise scoring. Calibration measures a vertical gap between claimed confidence and observed accuracy, with no direction on any response feature at all.

A biased judge can be well calibrated on the population it was biased on, because its confidence honestly reflects how often it produces the verdict it is disposed to produce; the curve comes out clean while the ranking underneath is wrong. A well-calibrated, unbiased judge can still slip once the effective standard moves under it, which is a rubric drifting across a run. Run the bias gates in their documented order before you read a calibration table, since a pairwise judge that fails a swap test attaches confidence to verdicts that were never stable. Both readings belong in the same judge report, and the per-bias taxonomy with the test that catches each one is laid out in the pillar.

Two different jobs go by the name calibration

Instrument calibration, in the sense a metrology lab means it, is comparing a device against a reference standard and adjusting it until it reads true, with a certificate and a recalibration interval attached. That meaning is legitimate and far older than this field, and it carries a discipline eval work usually skips: a calibration certificate names the conditions it holds under, and it expires.

Inside LLM evaluation the word does a second job as well. Calibrating a judge frequently means measuring its sensitivity and specificity against a human-labeled set and correcting the raw pass rate it produced, the Rogan-Gladen adjustment that the judge bias correction calculator implements. That procedure belongs to another page: the recipe for bias-correcting a judge-scored eval before reporting it owns it, including how to carry both the test-set and the calibration-set uncertainty into one interval.

The premise worth refusing is that either sense yields a portable certificate. A judge calibrated on support transcripts is not calibrated on code review, because the mapping from confidence to accuracy was fitted on an item distribution that no longer holds. The same goes for last quarter’s judge version against this quarter’s. Dropping low-confidence verdicts before computing a headline rate makes it worse, since the surviving set is no longer the set you sampled, which is coverage conditioning working on your judge instead of your model.

Report three things together: the calibration figure with its per-band counts, the item distribution it was fitted on, and the date it was measured, re-measured whenever the judge version, the rubric or the item mix moves.

Judge calibration is one of six gates on the pre-flight list a green eval skips, each carrying a pass criterion and a calculator, and confident wrongness has its own explainer for the non-specialist account of why a wrong answer sounds exactly like a right one. Agreement covers the label and calibration covers the confidence printed beside it, so a report quoting one without the other has characterized half an instrument, and the bias tests are owed before either reading is worth trusting. A calibration figure with no date on it is already out of date.