Topic
Judge reliability
Using a model to score another model: bias, position effects, agreement with humans, and how far a judge can be trusted.
Research
- Analysis
LLM judge bias vocabulary: six terms and their tests
Six terms name what goes wrong with an LLM judge: position bias, verbosity bias, self-preference, calibration, rubric drift, and the pairwise or pointwise choice beneath them. Each has its own test.
- Analysis
The LLM-judge bias checklist that gates your ranking
The LLM-judge bias checklist is eight pass/fail gates you run before trusting a ranking. Each gate pairs a detection test with a numeric pass line and the calculator that computes it.
- Analysis
Bias-correct your LLM-as-a-judge eval before reporting it
An LLM judge is an imperfect classifier, so its raw pass rate is biased. Correct it with the judge's sensitivity and specificity, then report a calibration-aware confidence interval.
- Analysis
LLM-as-a-judge bias, and the tests that catch it
LLM-as-a-judge bias is systematic, measurable distortion in an evaluator. A per-bias map pairs each bias with a detection test and a correction, so you can tell when a judge's ranking would flip.
- Analysis
Is your LLM-as-a-judge reliable? Test the evaluator
An LLM-as-a-judge is a fallible evaluator. Its reliability breaks along three axes, agreement, calibration, and bias, each with a test and a correction. This hub routes to all three.
Terms
-
Cohen's kappa (eval agreement)
Cohen's kappa is the chance-corrected agreement between exactly two raters labeling the same items into unordered categories: observed agreement less the agreement their own marginal rates would produce by accident, rescaled by the room chance left. In evals, a judge against a human reference.
-
Judge calibration (LLM evals)
Judge calibration is the correspondence between the confidence an LLM judge attaches to a verdict and how often verdicts carrying that confidence turn out correct, measured against held-out human labels rather than assumed from the judge's own scores.
-
Krippendorff's alpha (eval agreement)
Krippendorff's alpha is the proportion by which a set of labels falls below the disagreement chance would have produced, computed for any number of raters, on any measurement scale, with missing labels tolerated. Eval teams use it for judge-against-human agreement on graded rubrics.
-
Pairwise vs pointwise scoring
Pairwise and pointwise scoring are the two protocols an LLM judge can run under: pointwise grades one response on its own against an absolute scale, pairwise shows the judge two responses to the same prompt and asks which wins. Each licenses a different decision.
-
Position bias (LLM judges)
Position bias is an LLM judge's preference for whichever answer occupies a given slot in the prompt, first or last, independent of what the answer says. Measure it by scoring each pair in both orders and reporting the share of verdicts that survive the swap, with an interval.
-
Rubric drift (LLM judges)
Rubric drift is the movement of an LLM judge's effective scoring standard while the rubric text it is sent stays fixed, so two scores produced under the same rubric no longer sit on the same scale. The instrument changed between the measurements.
-
Self-preference bias (LLM judges)
Self-preference bias is an LLM judge scoring its own model family's responses above the score a reference standard gives the same text, measured as the gap between the judge's own-family win rate and the rate human annotators or a cross-family panel assign to identical pairs.
-
Verbosity bias (LLM judges)
Verbosity bias is an LLM judge's tendency to score a longer answer above a shorter one of equal or better quality, so response length acts as a scoring feature no rubric named. It is measured by padding an answer with words that add no information and counting the judge's flips.
Calculators
- Calculator
Inter-Rater Reliability Calculator (Cohen's Kappa)
Measure how well an LLM judge agrees with a human reference on pass/fail labels: Cohen's kappa, the chance-corrected agreement score, from a 2x2 table with its confidence interval and reading.
- Calculator
LLM Judge Bias Correction Calculator
Correct an LLM judge's raw pass rate for its measured sensitivity and specificity (the Rogan-Gladen correction), then report a calibration-aware confidence interval on the true rate.
- Calculator
LLM Judge Position Bias Calculator (Swap-Consistency Test)
Score a pairwise LLM judge on order-swapped pairs: swap-consistency and position-preference rates with Wilson intervals and an exact test for position bias.