LLM Judge Bias Correction Calculator

An LLM judge is an imperfect classifier, so its raw pass rate is a biased estimate of the true rate. This tool runs the same procedure worked end to end in Reporting LLM-as-a-judge evaluations: correct the raw rate with the judge's measured sensitivity and specificity, then attach an interval that carries both the test-set and the calibration-set uncertainty.

The correction. From the calibration set you get sensitivity Se = TP / (TP + FN) (the pass-class hit rate) and specificity Sp = TN / (TN + FP) (the fail-class hit rate). The judge says pass at rate p = c / n. Inverting the linear relationship between the observed rate and the true rate gives the Rogan-Gladen estimate θ̂ = (p + Sp - 1) / (Se + Sp - 1). The denominator is Youden's J, the judge's informativeness. It has to be positive (the judge beats a coin) or the correction is undefined; the smaller it is, the harder the judge amplifies every source of error.

The interval. The corrected rate depends on three estimated quantities, each with its own sampling error: the naive rate from n test items, and the two calibration rates from the human labels. Propagating all three through θ̂ with the first-order delta method gives Var(θ̂) ≈ [ Var(p) + (1-θ̂)²·Var(Sp) + θ̂²·Var(Se) ] / J², with Var(p) = p(1-p)/n, Var(Se) = Se(1-Se)/nSe, and Var(Sp) = Sp(1-Sp)/nSp. The eval-CI is θ̂ ± z·√Var(θ̂), clamped to [0, 1]. The naive rate keeps its own Wilson interval, which stays inside [0, 1] near the boundaries where the textbook normal interval does not.

Which method, and why. The delta-method interval is the one derived for LLM judges by Lee and colleagues and the one the host analysis reports, and it yields the additive variance split above, which tells you which calibration class to label next. It is a choice: the score-style Lang-Reiczigel interval has better small-sample coverage, but the delta interval already tends to overcover when calibration sets are small, so it runs conservative rather than optimistic. The tool surfaces that as a note instead of swapping the method.

Worked example (the defaults shown). With c = 700 of n = 1,000, 90 true positives and 10 false negatives on the pass class, and 85 true negatives and 15 false positives on the fail class: p = 0.70, Se = 0.90, Sp = 0.85, so J = 0.75 and θ̂ = (0.70 + 0.85 - 1) / 0.75 = 0.55 / 0.75 = 0.7333. The standard error is 0.0373, so the 95% eval-CI is [0.660, 0.807]. The naive Wilson interval is [0.671, 0.728]: the corrected interval is centered about three points higher and roughly two and a half times wider. The sensitivity term carries 61.7% of that width and the specificity term 11.6%, even though both classes got the same 100 labels, so the next human label is worth most on the pass class.

What the correction cannot fix. It removes only the bias you measure. It assumes the calibration labels are correct and that sensitivity and specificity are stable across the outputs you score, and it cannot touch a judge grading output from its own model family. Keep the judge independent of the system under test. The calibration set this page asks for is a sample of human-labeled items; a different sense of the word, judge calibration, asks whether the judge's stated confidence tracks how often it turns out to be right, and that one is scored in the Brier score and calibration error calculator.

Catch rate and pass accuracy. The catch rate is the share of real failures the judge flagged: items both your expert and the judge marked fail, over every item your expert marked fail. Pass accuracy is the share of real passes the judge passed: items both marked pass, over every item your expert marked pass. Each gets a Wilson score interval at the stated confidence level, two-sided.

Youden's J and its interval. J = catch rate + pass accuracy - 1. It runs from -1 (perfect disagreement) to 1 (perfect agreement) and equals zero when the judge is at chance. Its interval uses MOVER (the method of variance estimates recovery), which combines the two independent Wilson intervals, since the two classes are drawn from disjoint items.

At a stated failure rate. The page uses the production failure rate you enter, or the sample's share when the field is blank, and computes point values only, since that rate is an assumption. The judge fail rate is catch rate x failure rate + (1 - pass accuracy) x (1 - failure rate). The positive predictive value is the share of judge fails that are real failures; the negative predictive value is the share of judge passes that are real passes. Missed failures per 1,000 outputs are 1,000 x failure rate x (1 - catch rate), and false flags per 1,000 are 1,000 x (1 - failure rate) x (1 - pass accuracy).

Cohen's kappa. Raw agreement is the share of items where the judge and your expert gave the same label. Kappa adjusts for chance agreement: (observed - expected) / (1 - expected), where expected agreement uses the marginal proportions. The band is from the Landis and Koch scale, printed with its own source line.

Stratified labels. When an inclusionProbability column holds more than one value, each row is weighted by 1 / inclusionProbability and the page shows weighted point values only. Intervals for a sample drawn at unequal rates need a stratified method, which this scorecard leaves out.

Judge version comparison. The overall 2x2 counts items both versions got right, only the baseline got right, only the candidate got right, and neither got right. Overall accuracy for each version is its correct count over the total. The difference and its interval use Newcombe's method 10 for paired proportions, two-sided, since both judge versions grade the same items.

The test. The McNemar test asks whether the items only the candidate got right outnumber the items only the baseline got right, or the reverse, by more than chance would produce. It is printed in two forms, mid-p and exact. The mid-p form holds its stated error rate where the exact conditional form runs conservative (Fagerland, Lydersen and Laake 2013). A verdict rests on the interval; when the test disagrees, one line says so.

Per-class changes. The same paired difference and mid-p are computed within each class separately. These are unadjusted for multiplicity and labeled per class.

Attenuation. The observed difference between two versions equals Youden's J times the true difference, exactly when both versions share the same sensitivity and specificity. Dividing by J recovers the corrected difference. The items multiplier is at least 1/J squared: this judge needs at least that many times the items a perfect judge would need to see the same true difference.

The corrected-difference interval. The variance of the corrected difference carries both test-set and calibration-set error through the delta method: Var = Var(observed) / J^2 + observed^2 Var(J) / J^4, where Var(J) = catch rate x (1 - catch rate) / failures + pass accuracy x (1 - pass accuracy) / passes. Var(observed) depends on the design: pA(1 - pA)/nA + pB(1 - pB)/nB for separate items, and [(b + c)/n - ((c - b)/n)^2]/n for the same items, where b and c count the items the judge passed for only one version. The interval is left unclamped, since a true difference can sit anywhere from -1 to 1.

The per-version check. When calibration is per version, the page computes J for each version with its own labels, then tests whether J differs using a MOVER difference on their two intervals. When the interval excludes zero, one J would bias the corrected difference; each version's rate is corrected separately with the page's Rogan-Gladen correction, and the difference of the two corrected rates gets a MOVER interval on the two eval-CIs.

The lower-bound warning. When J is positive and above the weak-judge threshold, but its interval reaches zero or below at the stated confidence level, the direction of the correction is not established. The warning fires on both the single-rate correction and the two-version comparison, and it is suppressed when the page already says the correction is undefined or the judge is weak.

Formula: θ̂ = (p + Sp - 1) / (Se + Sp - 1); eval-CI = θ̂ ± z·SE

See Sources below