Topic
Judge reliability
Using a model to score another model: bias, position effects, agreement with humans, and how far a judge can be trusted.
Research
- Analysis
Kappa thresholds for LLM judges, and who published each one
Five published kappa bands from four sources, side by side, each with the author who wrote it, the date we read it, and the coefficient it was written for. They are conventions, and they disagree.
- Analysis
LLM judge bias vocabulary: the terms and their tests
Terms that name what goes wrong with an LLM judge: position bias, verbosity bias, self-preference, calibration, rubric drift, and the pairwise or pointwise choice beneath them. Defined, and placed.
- Analysis
The LLM-judge bias checklist that gates your ranking
The LLM-judge bias checklist is eight pass/fail gates you run before trusting a ranking. Each gate pairs a detection test with a numeric pass line and the calculator that computes it.
- Analysis
Bias-correct your LLM-as-a-judge eval before reporting it
An LLM judge is an imperfect classifier, so its raw pass rate is biased. Correct it with the judge's sensitivity and specificity, then report a calibration-aware confidence interval.
- Analysis
LLM-as-a-judge bias, and the tests that catch it
LLM-as-a-judge bias is systematic, measurable distortion in an evaluator. A per-bias map pairs each bias with a detection test and a correction, so you can tell when a judge's ranking would flip.
- Analysis
Is your LLM-as-a-judge reliable? Test the evaluator
An LLM-as-a-judge is a fallible evaluator. Its reliability breaks along three axes, agreement, calibration, and bias, each with a test and a correction. This hub routes to all three.
Terms
-
Cohen's kappa (eval agreement)
Cohen's kappa is the chance-corrected agreement between exactly two raters labeling the same items into unordered categories: observed agreement less the agreement their own marginal rates would produce by accident, rescaled by the room chance left. In evals, a judge against a human reference.
-
Judge calibration (LLM evals)
Judge calibration is the correspondence between the confidence an LLM judge attaches to a verdict and how often verdicts carrying that confidence turn out correct, measured against held-out human labels rather than assumed from the judge's own scores.
-
Krippendorff's alpha (eval agreement)
Krippendorff's alpha is the proportion by which a set of labels falls below the disagreement chance would have produced, computed for any number of raters, on any measurement scale, with missing labels tolerated. Eval teams use it for judge-against-human agreement on graded rubrics.
-
Pairwise vs pointwise scoring
Pairwise and pointwise scoring are the two protocols an LLM judge can run under: pointwise grades one response on its own against an absolute scale, pairwise shows the judge two responses to the same prompt and asks which wins. Each licenses a different decision.
-
Position bias (LLM judges)
Position bias is an LLM judge's preference for whichever answer occupies a given slot in the prompt, first or last, independent of what the answer says. Measure it by scoring each pair in both orders and reporting the share of verdicts that survive the swap, with an interval.
-
Rubric drift (LLM judges)
Rubric drift is the movement of an LLM judge's effective scoring standard while the rubric text it is sent stays fixed, so two scores produced under the same rubric no longer sit on the same scale. The instrument changed between the measurements.
-
Self-preference bias (LLM judges)
Self-preference bias is an LLM judge scoring its own model family's responses above the score a reference standard gives the same text, measured as the gap between the judge's own-family win rate and the rate human annotators or a cross-family panel assign to identical pairs.
-
Verbosity bias (LLM judges)
Verbosity bias is an LLM judge's tendency to score a longer answer above a shorter one of equal or better quality, so response length acts as a scoring feature no rubric named. It is measured by padding an answer with words that add no information and counting the judge's flips.
Calculators
- Calculator
Cohen's Kappa Calculator (Two Raters): Inter-Rater Reliability
Cohen's kappa from the four counts of a 2x2 table: chance-corrected agreement between two raters, with a confidence interval and an interpretation band. Built for an LLM judge against human labels.
- Calculator
LLM Judge Bias Correction Calculator
Correct an LLM judge's raw pass rate for its measured sensitivity and specificity (the Rogan-Gladen correction), then report a calibration-aware confidence interval on the true rate.
- Calculator
LLM Judge Position Bias Calculator (Swap-Consistency Test)
Score a pairwise LLM judge on order-swapped pairs: swap-consistency and position-preference rates with Wilson intervals and an exact test for position bias.
- Calculator
LLM Judge Calibration Calculator: Brier Score and ECE
Score an LLM judge's stated confidence against observed correctness: Brier score, expected and maximum calibration error, a reliability diagram, and the error rate behind its confident verdicts.
- Calculator
Pairwise Win Rate Calculator with Elo and Bradley-Terry Ratings
Turn pairwise LLM judge comparisons between models or prompts into win rates with Wilson intervals, Bradley-Terry strengths on the Elo scale, and a plain read on whether the advantage is real.
- Calculator
Fleiss' Kappa, Krippendorff's Alpha and ICC Calculator
Agreement among three or more raters or LLM judges: Fleiss' kappa, Krippendorff's alpha, Gwet's AC1 and intraclass correlation, with percent agreement, bootstrap intervals, and per-rater diagnostics.
- Calculator
Your judge disagrees with your humans. Which bias explains it?
Paste judge and human labels. See which bias mode explains the gap, ranked by how much each one accounts for, then follow the link to the calculator that tests it.