INSTRUMENT | eval
Judge Validation Report Builder: Report, Version Card, Drift Plan
2 cited sources
Fill one form and keep three documents: a validation report showing your LLM judge was checked against people, a version card pinning the prompt and settings, and a drift plan with a date on it.
A judge is worth citing only if someone checked it against people first. Fill this in once and keep three documents: a validation report, a version card pinning the prompt and settings behind your number, and a drift plan with a date on it. It rests on our work on judge reliability and our guide to reporting judged evaluations.
Score a judge's stated confidence with the calibration calculator, and check order effects with position bias.
Showing your last valid result. Update the inputs above to recompute.
Cohen's kappa
0.65
content-safety-judge v3: Cohen's kappa 0.65 against adjudicated consensus from 2 annotators on 200 items; next drift check 2026-09-21
Derived from the two by two counts, with the human baseline as truth.
| Measure | Share | Out of |
|---|---|---|
| Raw agreement | 90.0% | all items |
| Catch rate | 80.0% | real failures |
| False escalation rate | 8.2% | real passes |
| Flag precision | 63.2% | items the judge flagged |
| Baseline prevalence | 15.0% | all items |
The two by two counts were left empty, so no rates are derived here. The agreement figure above stands on its own.
Drift plan
- Manifest fingerprint
- computing
- Next drift check
- computing
- Fresh-sample margin
- computing
Validation report
Judge prompt and version card
Drift canary plan
catch rate = tp / (tp + fn) · margin = 1.96 × √(0.25 / n)How?
How this is calculated
The five rates. They run on the optional two by two block, with the human baseline as truth and the judge as the test. Raw agreement is the share of items the two agreed on. Catch rate is the share of real failures the judge flagged. False escalation is the share of good items it flagged anyway. Flag precision is the share of its flags that were real. Baseline prevalence is the share of the set that were real failures. Together they size the human-review budget the review threshold optimizer allocates.
Why prevalence matters more than it looks. With 24, 14, 6 and 156, raw agreement reads 90.0 percent and prevalence reads 15.0 percent. On a rare failure a judge that passes everything still scores high, which is why the number worth reporting is a chance-corrected coefficient, computed elsewhere. What value of that coefficient is good enough is a separate question. The published bands disagree with each other, and each was written for a particular statistic, so the table you read your number against moves the verdict.
These five rates are point values. They carry no interval of their own. For an interval on a rate, use the pass rate confidence interval calculator. To ask whether the judge flags at a different rate than the human baseline, rather than just by how much, the same two by two feeds the McNemar test calculator.
The fresh-sample margin. A proportion's spread is widest at one half, so 1.96 times the square root of 0.25 over n is the worst a 95 percent interval gets at that sample size. Fifty items gives plus or minus 13.9 points. A hundred gives 9.8. One gives 98.0, which is the whole scale and is the honest answer for a set of one.
The next due date. The chosen cadence carries a machine-readable period, and the date is that period added to the pin date. A month is a calendar month clamped at the end of a shorter one: 31 January plus a month is 28 February, never 2 March. Where a source gives a range, the earlier end sets the date and the far end prints beside it. Where a source names a trigger rather than an interval, no date is computed and the plan says so.
The alarm rule. Two clauses, and either one fires. A case fires when its re-run verdict differs from its own recorded baseline verdict. A case fires when it disagrees with itself across the repeat runs. The baseline match count is reported, never used as a third trigger, because a drop in it is already covered by the first clause. There is no probability anywhere in the plan: two dozen adversarial items cannot carry a significance claim.
The two fingerprints. The card fingerprint is a SHA-256 over every setting that can move a verdict, printed as twelve hex characters. The prompt fingerprint covers the prompt text alone, as eight. Two cards differing only in top-p print different card fingerprints and the same prompt fingerprint, which is how a reader sees which half changed. Renaming the judge changes neither. Windows line endings and trailing spaces are normalized first, so the same prompt pasted from two machines is one card.
Formula: catch rate = tp / (tp + fn) · margin = 1.96 × √(0.25 / n)
What published sources say about re-checking
Three sources, and they disagree. That spread is the finding, so all of it is shown rather than averaged into one number this page would then have to defend. Each row carries the name of who published it and the day a person here read it.
Calibration and drift cadences as published, with the date each was read.
| What the source schedules | Set size | How often | Source | Read on |
|---|---|---|---|---|
| Initial calibration, dual labeled | 200 | once | Future AGI | 2026-08-21 |
| Drift check, fresh set | 50 | monthly | Future AGI | 2026-08-21 |
| Alert on rolling kappa drop | none stated | 2 point rolling drop, per model version change | Future AGI | 2026-08-21 |
| Annotations per error code | 15 to 20 | per calibration round | AWS sample-GEDD | 2026-08-21 |
| Recalibration | fresh annotation set | quarterly, every 60 to 90 days | AWS sample-GEDD | 2026-08-21 |
| Production trace sample | 100 to 300 | per calibration loop | Future AGI | 2026-08-21 |
| Canary eval against fixed ground truth | none stated | weekly | Galileo | 2026-08-21 |
| Human spot check, stratified | none stated | monthly | Galileo | 2026-08-21 |
| Full calibration cycle with SME annotation | none stated | quarterly, or on a red-flag signal | Galileo | 2026-08-21 |
Questions
Questions
Why is raw agreement not enough?
Because two labelers who both guess can agree most of the time when one label is rare. The worked example on this page is the mild version: 90.0 percent agreement on a set where 15.0 percent of items are real failures. The severe version is a set where 95 of 100 items are safe, a judge that says safe every time scores 90 percent, and a chance-corrected coefficient comes out near zero or below it. That coefficient is the number worth reporting, and this page asks you to bring it rather than computing it.
How big should the canary set be?
This page names no size of its own. The nearest published floor is 15 to 20 annotated responses per error code (AWS sample-GEDD, read 2026-08-21), and it is worth saying plainly that AWS wrote that for kappa annotation sets and never mentions canary sets at all. So treat it as a floor to clear, not a target to hit. The rule that does transfer: the set covers every failure mode you named above, and it stays frozen. The worked example ships 24 cases because that is what fits twelve failure modes twice, not because 24 is right.
The cadence sources disagree. Which one do I use?
Pick the one whose scope is closest to your situation and say in the report which one you picked. Monthly, every 60 to 90 days and quarterly are all published positions, and the table above shows each with its source and read date. If none of them fits, choose your own interval in days; the plan then says the interval was chosen by hand rather than taken from a source.
Is the version card a schema?
Yes. The card exports against a published JSON Schema at
/schemas/judge-card/v1.json. The version lives in the path and a published version
is never changed in place: fields may be added inside v1, and anything that changes what an
existing field means publishes a v2 beside it. That is what lets a card written today still
read correctly in two years.
What happens to a document I load back in?
The JSON export carries the whole form, so loading one back fills every field. A file written by a later minor version still loads, and any field this page does not recognize is listed, kept, and written back out when you export again. A file from a higher major version is refused with both versions named, because a major bump can change what a field means and there is no way to tell from the file. A plan handed to the card control is refused too, by name.
Can I leave fields blank?
Yes, and the export still works. Every blank required field is listed above the export bar and prints as NOT RECORDED in the document. A visible gap is better than an invisible one: a reader of your report can see what was not measured, which is exactly what a quietly dropped line would hide.