INSTRUMENT · EVAL
LLM Judge Bias Correction Calculator
Correct an LLM judge's raw pass rate for its measured sensitivity and specificity, then report a calibration-aware confidence interval on the true rate.
Bias-corrected pass rate
73.3%
95% eval-CI [66.0%, 80.7%]
What the judge reported versus what the correction recovers.
| Estimate | Rate | Interval |
|---|---|---|
| Naive judge rate | 70.0% | [67.1%, 72.8%] |
| Bias-corrected rate | 73.3% | [66.0%, 80.7%] |
Judge diagnostics
- Sensitivity (Se) judge catches a real pass
- 90.0%
- Specificity (Sp) judge catches a real fail
- 85.0%
- Youden's J Se + Sp - 1, must be > 0
- 0.750
- Bias removed corrected minus naive
- +3.3 pp
- Calibration labels pass class / fail class
- 100 / 100
The correction divides every source of error by Youden's J, so a weaker judge widens all three.
| Where the width comes from | Share of variance |
|---|---|
| Test set (n items) | 27% |
| Specificity (fail-class labels) | 12% |
| Sensitivity (pass-class labels) | 62% |
The pass-class calibration labels drive the width. Spend your next human labels there.
θ̂ = (p + Sp - 1) / (Se + Sp - 1); eval-CI = θ̂ ± z·SEHow?
How this is calculated
An LLM judge is an imperfect classifier, so its raw pass rate is a biased estimate of the true rate. This tool runs the same procedure worked end to end in Reporting LLM-as-a-judge evaluations: correct the raw rate with the judge's measured sensitivity and specificity, then attach an interval that carries both the test-set and the calibration-set uncertainty.
The correction. From the calibration set you get
sensitivity Se = TP / (TP + FN) (the pass-class hit rate) and
specificity Sp = TN / (TN + FP) (the fail-class hit rate). The
judge says pass at rate p = c / n. Inverting the linear
relationship between the observed rate and the true rate gives the
Rogan-Gladen estimate θ̂ = (p + Sp - 1) / (Se + Sp - 1). The
denominator is Youden's J, the judge's informativeness. It has to be
positive (the judge beats a coin) or the correction is undefined; the
smaller it is, the harder the judge amplifies every source of error.
The interval. The corrected rate depends on three
estimated quantities, each with its own sampling error: the naive rate from
n test items, and the two calibration rates from the human labels.
Propagating all three through θ̂ with the first-order delta method gives
Var(θ̂) ≈ [ Var(p) + (1-θ̂)²·Var(Sp) + θ̂²·Var(Se) ] / J², with Var(p) = p(1-p)/n, Var(Se) = Se(1-Se)/nSe,
and Var(Sp) = Sp(1-Sp)/nSp. The eval-CI is
θ̂ ± z·√Var(θ̂), clamped to [0, 1]. The naive rate keeps its
own Wilson interval, which stays inside [0, 1] near the boundaries where the
textbook normal interval does not.
Which method, and why. The delta-method interval is the one derived for LLM judges by Lee and colleagues and the one the host analysis reports, and it yields the additive variance split above, which tells you which calibration class to label next. It is a choice: the score-style Lang-Reiczigel interval has better small-sample coverage, but the delta interval already tends to overcover when calibration sets are small, so it runs conservative rather than optimistic. The tool surfaces that as a note instead of swapping the method.
Worked example (the defaults shown). With c = 700 of n = 1,000, 90 true positives and 10 false negatives on the pass class, and 85 true negatives and 15 false positives on the fail class: p = 0.70, Se = 0.90, Sp = 0.85, so J = 0.75 and θ̂ = (0.70 + 0.85 - 1) / 0.75 = 0.55 / 0.75 = 0.7333. The standard error is 0.0373, so the 95% eval-CI is [0.660, 0.807]. The naive Wilson interval is [0.671, 0.728]: the corrected interval is centered about three points higher and roughly two and a half times wider. The sensitivity term carries 61.7% of that width and the specificity term 11.6%, even though both classes got the same 100 labels, so the next human label is worth most on the pass class.
What the correction cannot fix. It removes only the bias you measure. It assumes the calibration labels are correct and that sensitivity and specificity are stable across the outputs you score, and it cannot touch a judge grading output from its own model family. Keep the judge independent of the system under test.
Formula: θ̂ = (p + Sp - 1) / (Se + Sp - 1); eval-CI = θ̂ ± z·SE
Questions
Why not just report the judge's raw pass rate?
Because the judge is a classifier with its own error rates, its raw pass rate estimates the classifier's behavior, and a release gate reads it as the model's accuracy. The two differ by a bias you can measure from a small human-labeled calibration set and remove. On the defaults, the raw 70.0% understates a corrected 73.3%; more test data narrows the raw interval but never removes that gap.
How many human labels do I need in each calibration class?
Enough that Youden's J is comfortably positive and the interval is usable, then allocated by the variance split, not evenly. This tool shows each class's share of the interval width; spend your next labels on the class with the larger share. Fewer than about 30 labels in a class triggers the overcoverage note, because the Rogan-Gladen interval runs conservative on small calibration sets (Chen et al., preprint, cited below).
What does an undefined correction mean?
Youden's J is sensitivity plus specificity minus one. When it is zero or negative the judge is at or below chance, the inversion divides by a non-positive number, and there is no valid corrected rate. The fix is a better judge or more calibration labels, not a different formula.
Why is my corrected estimate pinned at 0% or 100%?
The Rogan-Gladen estimate can fall outside [0, 1] when the naive rate is inconsistent with the calibrated error rates, most often on a small calibration set. The tool clamps the estimate and any interval bound back into [0, 1] and flags it, since the delta approximation is already unreliable at the boundary.
Is reliability@k or pass^k the same thing?
No. Those are the lane's terms for aggregating pass or fail outcomes across a task suite. The corrected per-task rate here is the honest input to that aggregation: feed it uncorrected judge labels and the same bias enters the aggregate one task at a time. Correct first, aggregate second.
Sources
- Rogan & Gladen (1978), Estimating prevalence from the results of a screening test
- Lee et al. (2026), How to Correctly Report LLM-as-a-Judge Evaluations (ICML 2026)
- Chen et al. (2026), Efficient Inference for Noisy LLM-as-a-Judge Evaluation (preprint)
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.2.4.1