LLM Judge Calibration Calculator: Brier Score and ECE

Each judged item pairs the judge's stated confidence with whether the verdict was actually correct against ground truth. Calibration asks whether those stated probabilities carry information: among items the judge marked near 0.9, roughly 90% should be correct. A judge can be accurate and badly calibrated at the same time, and a well-calibrated judge can still be weak, so the two properties deserve separate measurement. Calibration is one test in the wider question of whether a model can be trusted to grade at all, which is worked through in testing whether an LLM-as-a-judge is reliable.

Brier score. The mean of (confidence minus outcome) squared across all items (Brier 1950). It is a strictly proper scoring rule and needs no binning, which makes it the one number on this page that no binning choice can move. Lower is better. A judge that always says 0.5 scores exactly 0.25 on any data. The sharper benchmark on skewed data is the uncertainty term in the decomposition table below: a judge that always states the overall base rate scores exactly that value, so a Brier score above the uncertainty term means the stated confidences carry less information than the base rate alone.

ECE and MCE, and what binning hides. Items are grouped into confidence bins. Each occupied bin contributes its count-weighted absolute gap between mean stated confidence and observed accuracy; ECE is the sum of those contributions (Naeini et al. 2015, in the formulation of Guo et al. 2017), and MCE is the largest single-bin gap. Both move with the binning: more bins expose structure but thin the per-bin counts, and a wide bin can hide offsetting errors, because overconfident and underconfident items inside one bin cancel. The default is 10 equal-width bins, the convention most of the literature reports. Equal-mass bins instead hold per-bin counts steady, which helps when confidences cluster near the top of the range, as LLM judge confidences usually do. Interior bin edges belong to the upper bin, and a confidence of exactly 1 lands in the last bin. In equal-mass mode the table's bin ranges are the observed minimum and maximum confidence in each bin rather than nominal edges, and items with identical confidence can split across adjacent bins to keep the counts even.

Murphy decomposition. On the same bins, the Brier score splits into reliability (the miscalibration penalty, lower is better), resolution (how far per-bin accuracy moves away from the base rate, higher is better), and uncertainty (the base rate's own variance, fixed by the data). The textbook identity, reliability minus resolution plus uncertainty, holds exactly only when every item in a bin shares one confidence value, so the table reports the leftover within-bin residual explicitly rather than letting the three terms quietly fail to add up to the Brier score.

Per-bin intervals and high-confidence errors. Each bin's accuracy carries a Wilson 95% interval, because a gap computed on a five-item bin is mostly noise and the interval says so. The high-confidence slice counts items at or above the threshold (inclusive, default 0.90) and reports how many of those confident verdicts were wrong. That error rate is the number to check before auto-accepting a judge's confident verdicts.

Worked example. The preloaded 24 items score 18 correct (75.0% accuracy) against a mean stated confidence of 82.8%, an overall overconfidence of 7.8 points. With 10 equal-width bins, ECE is 8.2% and the Brier score 0.185. The top bin holds the ten most confident verdicts at a mean stated confidence of 95.4% against 80.0% observed accuracy, a 15.4-point gap that is also the MCE, and 2 of the 10 verdicts at or above 0.90 confidence are wrong. Rerunning the same rows with 5 equal-mass bins moves ECE to 9.7% and MCE to 26.6%, which is the binning dependence described above at work, and the reason this page always names its binning next to the number.

Honest limits. Every figure is arithmetic on the rows you paste. The correctness column is itself a measurement, and a mislabeled reference set skews a calibration read in either direction. Stated LLM confidences are generated text, and a verbalized 0.9 is under no obligation to behave like an internal probability, which is precisely why measuring it is worth the trouble, and what makes a model sound sure when it is wrong is the plain-language account of why stated confidence and correctness come apart. Quote ECE with its bin count and strategy attached, and compare judges only under the same scheme. The method for carrying a judge's measured error rates into the eval score you report is worked through in bias-correcting judge evals before reporting them.

Formula: Brier = mean (confidence - outcome)^2; ECE = Σ (n_b / N) |conf_b - acc_b|

See Sources below