The LLM-judge bias checklist that gates your ranking

Most LLM-judge audits are a single swap test, run once and eyeballed. The bias that bends a ranking hardest is usually the gate that check never reaches: a systematic study across five judge models found formatting the largest bias when content was held constant, above the answer-order bias most audits fixate on (Soumik, 2026, as of 2026-07).¹ A judge is only as trustworthy as the gate you skipped, and the skipped gate is rarely the famous one.

The cost lands at the release gate, which reads a judge’s raw pass rate as the model’s accuracy and lets whichever distortion you left unmeasured decide the close calls. A checklist clears a judge only when every item on it carries a number you can fail against.

Before you trust an LLM judge’s ranking, clear one gate per distortion in cost order, each with a numeric pass line and the calculator that computes it: a reference exists to be biased against, agreement clears a Cohen’s kappa of 0.61, and the ranking survives an order swap, a length match, an identity mask, a cue strip, and a format normalize, then the reported rate is bias-corrected with its interval. Any gate that moves a decisive pair sends that pair to a jury or a human. Skip a gate and that axis ships on an assumption.

This is the operational companion to the per-bias detection map: that page explains why each distortion exists and how each test works; this one is the run-order, the pass line, and the tool for every gate.

What does an LLM-as-a-judge bias checklist check?

The LLM-as-a-judge bias checklist is eight gates you run before trusting a ranking, ordered cheapest-first. Each gate names one distortion, a procedure that exposes it, a numeric pass line, and the calculator that computes that line. A ranking ships only when every gate its task exposes passes, or the pairs that moved escalate to a jury or a human.

The gates split into three tiers by cost, and the rest of this page is that division: the three you run on every judge, the four you run only when your task exposes them, and the reporting gate that closes the audit.

Run the cheap gates on every judge

Gates 0 through 2 are cheap enough that there is no task where you skip them. They cost a small labeled subset and one extra pass of the judge, and they catch the failures that invalidate everything downstream.

Gate 0: is there a reference to be biased against?

This is the precondition the other seven forget. A preference counts as bias only against a reference the judge is meant to reproduce, so assemble that reference first: a human-labeled subset, or a verifiable ground truth like a unit test or an exact-match key. Without one, a judge that rewards longer answers might be biased or might just hold a defensible preference, and nothing lets you separate the two. When the whole evaluation is verifiable, the judge stops being the source of truth and most gates below never reach the score.

Gate 1: does the judge agree past chance?

Chance-corrected agreement is the bar, because raw agreement inflates whenever one answer is obviously better and both graders pocket the easy calls for free. Score the calibration subset with the judge and a human, then compute a chance-corrected coefficient instead of a bare match rate. The pass line comes from the inter-rater reliability calculator: a Cohen’s kappa of 0.61 or higher, which the Landis and Koch scale reads as substantial agreement, with 0.81 and up as almost perfect (Landis and Koch, 1977, as of 2026-07).² Report the kappa with its interval; a point estimate near the threshold with a wide interval has not cleared it.

A judge below substantial agreement is scoring a different construct than the one you asked for, a construct-validity failure that no bias correction can fix.

Gate 2: does the verdict survive an order swap?

Position is the base case and the cheapest bias to catch. Score each decisive pair as (A, B), then again as (B, A), and treat any disagreement as position bias rather than signal. The position bias calculator turns those swaps into a position-robustness rate, where a higher value means the order moved the judge less; across models that rate has run from about 0.57 to 0.83, reported without intervals and best read as directional (CALM, 2024, preprint, as of 2026-07).³ A strong number is no guarantee and a weak one is common, and most judges tilt toward whichever answer they see first, a lean the MT-Bench study found even in a judge that otherwise matched human preferences about 85 percent of the time, above the 81 percent humans reached with each other (Zheng et al., 2023, as of 2026-07; reported without intervals).⁴

The pass line is stricter than the average, because a ranking turns on its close pairs: put a pass-rate confidence interval on the flip rate and compare its upper bound to your smallest win margin.

Suppose 100 pairs are decisive and the judge flips on 8 of them. The flip rate is 0.08 and a Wilson interval runs roughly 0.04 to 0.15 (computed from the stated assumptions, not measured). If your tightest margin is five points, the interval’s upper bound exceeds it, so position bias can still decide a close pair and the gate fails.

For the paired verdicts specifically, a McNemar test on the discordant (A, B) versus (B, A) pairs tells you whether the swap-induced disagreement is more than sampling noise.

Run the task-specific gates on the biases your task exposes

Gates 3 through 6 cost more and only some bind on any given task. Pick the ones your evaluation actually exposes: open-ended generation exposes length and surface, a same-family judge exposes self-preference, factuality tasks expose authority cues. The per-bias map explains the mechanism behind each; here is the procedure and the number.

Gate 3: is the score paying for length?

A verbose answer can win on words alone. Build length-matched rewrites that hold content fixed, then score each decisive pair with the raw answer and again with its length-matched version. The correction with the strongest backing is length-control after the fact, which raised one public leaderboard’s human correlation from 0.94 to 0.98, a point pair the paper reports without intervals (Dubois et al., 2024, as of 2026-07).⁵ Because the two scorings are paired on the same items, a McNemar test returns both whether the length-flipped verdicts exceed sampling noise and a confidence interval on the raw-versus-length-controlled win-rate change: pass if that interval includes zero, and report the corrected number if it excludes zero.

Read the flip rate itself with a pass-rate confidence interval and confirm its upper bound sits below your smallest win margin. Regressing the score on answer length and reading whether the slope’s interval clears zero is a defensible alternative, but no calculator on this site computes that regression yet. On open-ended tasks this is often the gate that shifts a ranking most.

Gate 4: is the judge grading its own family?

Self-preference appears when the model that produced an answer also grades it, and it tracks how well the judge recognizes its own text (Panickssery et al., 2024, preprint, as of 2026-07).⁶ Mask provenance, then run a jury of judges from different model families over the same pairs. A McNemar test on same-family versus cross-family verdicts flags a judge that systematically favors its own lineage, and the inter-rater reliability calculator checks that the jury agrees past chance. This gate matters most when a judge is used to attribute a failure to the responsible step, because a judge that quietly clears its own family will pin the failure on the wrong agent.

Gate 5: does a planted cue move the verdict?

Bandwagon and authority are one gate: the judge moving on a signal planted in the text, a stated majority, a vote count, a fabricated citation, a credential. Inject a controlled false cue, add “90% of reviewers prefer A” to probe bandwagon or attach a made-up citation to the weaker answer to probe authority, and measure the flip rate with a pass-rate confidence interval. Pass if the interval’s upper bound sits below your smallest win margin; if it moves the verdict, strip the cue from the judge’s context. Retrieval-augmented judging is where this bites in production: the passages a judge retrieves come stamped with source names and assertive phrasing, smuggling an authority cue in alongside the actual evidence.

Gate 6: does surface format decide it?

This is the gate most audits skip and the one that bites hardest. Formatting was the largest measured bias across five judge models with content held constant, far exceeding position bias in the same study (Soumik, 2026, as of 2026-07).¹ Score identical content plain versus formatted, and neutral versus warm in tone, then read the flip rate with a pass-rate confidence interval and confirm the discordance with a McNemar test. Pass if the flip rate’s upper interval bound sits below your smallest win margin; if it exceeds that margin, normalize every candidate to the same format and a neutral register before judging, or write a substance-only rubric.

Correct the rate before you report it

The gates above defend a ranking; the last one defends the number you publish.

Gate 7: is the reported rate bias-corrected?

A judge is an imperfect classifier, so its raw pass rate is a biased estimate of true accuracy, off by an amount set by its own error rates, and more test items only converge more tightly onto the wrong number.

Run the judge-bias correction calculator on the sensitivity and specificity you already measured in Gate 1. It inverts the naive rate with the plug-in estimator and returns a bias-corrected accuracy plus an interval that carries both test-set and calibration-set uncertainty, the calibration-aware interval Lee and colleagues formalize for LLM judges (Lee et al., 2026, as of 2026-07).⁷ The pass line is procedural: never quote a judge’s raw pass rate as the model’s accuracy, and never quote the corrected rate without the wider interval that admits you measured the judge. The full arithmetic, from a calibration set to a corrected rate to where you spend your next human label, lives in the reporting procedure, and the interval itself is defined in the eval confidence interval glossary.

The calculators each gate links to are shipped tools you run yourself. The reliability profiler this site is building toward is designed to run this entire checklist against your own evaluator and report every correction with its confidence interval; it is pre-launch, so that is design intent, and no such measured number is claimed here.

Order the run, then triage what moved

Run the gates in the order they cost. Reference, agreement, and position first, on every judge, because they are cheap and they invalidate everything downstream when they fail. Then the task-specific gates your evaluation exposes. Then the reporting correction, last, on the rate that survived. The order matters because a judge that fails Gate 1 is scoring the wrong thing, and correcting a bias in a judge that does not agree with humans is polishing the wrong number.

Every gate produces the same output: a set of pairs where the verdict moved. Triage them by cost.

Most pairs are lopsided and survive every gate untouched, so they stay with the automated judge and the expensive review concentrates where the signal is weakest. A corrected per-task verdict is also the right input to any suite-level reliability aggregate, because a metric that reads each verdict as ground truth will inherit whatever bias you left in one task at a time.

That rule fits on an index card, and it is the whole method: confirm a reference, walk the gates your task exposes in cost order, correct what each one moves, and publish the ranking with its interval attached. A judge that survives every gate untouched is one you can cite for that comparison without an asterisk. A judge that trips one has just handed you the exact pair a lone agreement score would have graded wrong. For the terms behind the gates, see the reliability glossary; to route across all three judge-reliability axes, start at the LLM-as-a-judge hub; to watch these corrections run on real data, read the published methods.