LLM judge agreement and bias calculators
Check whether an LLM judge can be trusted: agreement with human labels for two or more raters, calibration of its confidence, position bias, bias-corrected pass rates, and pairwise win rates.
What these numbers mean
When a model grades another model’s work, the grade is itself a measurement, and it can be wrong in patterns. A judge can track your human labels on easy items and drift on hard ones, favor whichever answer it reads first, or sound most certain on the calls it gets wrong. The calculators here measure those patterns on labels you already have, correct a reported pass rate for a judge’s known error rates, and turn head-to-head verdicts into rankings.
| What you have | The instrument |
|---|---|
| A judge already grading and nothing written down about how it was checked | write the validation record |
| One judge and one human, pass or fail, on the same items | measure chance-corrected agreement |
| A judge that over-passes or over-fails | correct the reported pass rate |
| Pairwise verdicts, each pair also run swapped | test for order preference |
| A judge that states a confidence each time | check if the confidence is earned |
| Head-to-head comparisons between models | rank them by win rate |
| Three or more raters, or graded scores, on the same items | measure agreement across a panel |
| Two coefficients that disagree about the same labels | compare the two coefficients on the same data |
| An agreement coefficient computed over items that may repeat | count the duplicates first |
| A grader agreement review you have to pay for | price the review before it runs |
| A judge measured once, with no way to tell if it is still the same judge next month | log it and watch for drift |
These checks come out of our own judge analyses: testing the evaluator before you trust it, the biases a judge shows and the tests that catch them, and what a judge score owes a reader before you report it. Run the bias checklist before you publish a ranking.
The fork most people hit is how many raters they have. With two raters on the same items the statistic is Cohen’s kappa, which is what the agreement calculator computes. Three or more raters, or graded scores instead of pass and fail, needs Fleiss’ kappa, Krippendorff’s alpha, or an intraclass correlation. Running the two-rater number on a panel of three means picking two of them and throwing the third rater’s labels away.
The second fork is prevalence, and it trips people up because both numbers look right. When nearly every item passes, a chance-corrected coefficient can read moderate while AC1 reads 0.945 on the identical labels. The two statistics correct for chance in different ways, and a lopsided pass rate is what pulls them apart, so the gap is information about the set rather than a fault in either number. Which published bands are worth quoting, and where each one came from sets out which threshold to cite and when a disagreeing pair is the honest thing to report.
The number is not the end of it, either: the same judge, checked again next month, can read differently with nothing you did causing it, and the agreement tracker is where that reading goes to become a series instead of a one-off.
Tools in this topic
Judge Validation Report Builder: Report, Version Card, Drift Plan
Fill one form and keep three documents: a validation report showing your LLM judge was checked against people, a version card pinning the prompt and settings, and a drift plan with a date on it.
Instrument | LLM judge agreement and biasCohen's Kappa Calculator (Two Raters): Inter-Rater Reliability
Cohen's kappa from the four counts of a 2x2 table: chance-corrected agreement between two raters, with a confidence interval and an interpretation band. Built for an LLM judge against human labels.
Instrument | LLM judge agreement and biasLLM Judge Bias Correction Calculator
Correct an LLM judge's raw pass rate for its measured sensitivity and specificity (the Rogan-Gladen correction), then report a calibration-aware confidence interval on the true rate.
Instrument | LLM judge agreement and biasLLM Judge Position Bias Calculator (Swap-Consistency Test)
Score a pairwise LLM judge on order-swapped pairs: swap-consistency and position-preference rates with Wilson intervals and an exact test for position bias.
Instrument | LLM judge agreement and biasLLM Judge Calibration Calculator: Brier Score and ECE
Score an LLM judge's stated confidence against observed correctness: Brier score, expected and maximum calibration error, a reliability diagram, and the error rate behind its confident verdicts.
Instrument | LLM judge agreement and biasPairwise Win Rate Calculator with Elo and Bradley-Terry Ratings
Turn pairwise LLM judge comparisons between models or prompts into win rates with Wilson intervals, Bradley-Terry strengths on the Elo scale, and a plain read on whether the advantage is real.
Instrument | LLM judge agreement and biasFleiss' Kappa, Krippendorff's Alpha and ICC Calculator
Agreement among three or more raters or LLM judges: Fleiss' kappa, Krippendorff's alpha, Gwet's AC1 and intraclass correlation, with percent agreement, bootstrap intervals, and per-rater diagnostics.
Instrument | LLM judge agreement and biasJudge Agreement Tracker: Log Kappa Over Time
Keep a dated log of one judge's agreement with human labels: the value, the prompt version and set behind it, the published band it falls in, the change since the last run, and the next check date.
Analyses that use these calculators
- Judge reliability
Kappa thresholds for LLM judges, and who published each one
Five published kappa bands from four sources, side by side, each with the author who wrote it, the date we read it, and the coefficient it was written for. They are conventions, and they disagree.
- Multi-agent failures
AdaMAST: an adaptive taxonomy for multi-agent failures
AdaMAST induces a failure taxonomy from an agent system's own traces: named codes on three fixed axes, no hand-authored labels. What the paper measured, and how much of the gain adaptivity supplies.
- Eval statistics
Eval statistics: which number needs which test
The statistics an agent eval rests on, routed by the question in front of you: sizing before the run, the interval on a rate, a paired test on a delta, and agreement on the labels.
- Judge reliability
LLM judge bias vocabulary: six terms and their tests
Six terms name what goes wrong with an LLM judge: position bias, verbosity bias, self-preference, calibration, rubric drift, and the pairwise or pointwise choice beneath them. Each has its own test.
- RAG & retrieval
RAG evaluation metrics, and what each one certifies
Each RAG evaluation metric certifies one narrow property. What context precision, context recall, faithfulness, groundedness, answer relevancy and retrieval drift answer, and what each cannot see.