TOOL | Judge reliability
Build a labeling handbook from criteria, with rules, examples and a quiz
Write the rules your labelers follow so that their labels match yours: what each label covers, examples, which label wins when several fit, and what to do when they cannot decide. A quiz checks a new labeler first. You get the handbook, a quiz CSV, an answer key and a precedent log. No agreement threshold is preset, because published kappa bands disagree; the agreement calculators compute agreement.
Start from a rubric in the criteria builder, or type labels here.
The handbook, quiz and log below come from the returns-desk example, a fictional returns desk for an online outdoor-gear shop.
Reads a saved labeling handbook or a labeling precedent log, as a JSON file. Also reads a grading rubric from the criteria builder.
JSON. Drag one here, or use the box below.
Your file or pasted text is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.
Check this value.
Opening this file replaces the current handbook. Export it first if you want to keep it.
Load the example to see what a finished handbook looks like, or add your first criterion or label. You can also import a rubric from the criteria builder.
Checks
All checks pass. The checks look for missing parts, overlapping definitions and gaps in the decision order; whether each rule fits your task is for the people who review the handbook.
How this is calculated
The page builds a decision order from the precedence rules. Each rule adds an edge from one label to another. A label that can be reached from itself through those edges is in a cycle; the order is undefined until the cycle is resolved. When no cycle exists, the rules chain into one order, and the page prints it only when every pair is ordered. Pairs with no path in either direction are unordered; labelers escalate these to the escalation contact.
Two labels are flagged as overlapping when the content words their names and definitions share, counted once in each label, make up at least half of all the content words in the two (a Dice coefficient of 0.5 or above, after a closed list of stopwords is removed and trailing -s and -ly are trimmed).
The page orders quiz cases from the seed, and the same seed gives the same order for the same set of cases. The quiz CSV lists cases in that order with an empty answer column; the answer key CSV lists them with the correct answer and the rule tested.
The fingerprint is a SHA-256 hash of the handbook's task, graded unit, scheme, active labels (sorted by id), precedence rules (sorted), cannot-judge policy and escalation. Quiz cases, calibration settings, the title and the rubric reference are left out, so editing the quiz or the calibration plan does not re-version the handbook.
The version starts at 1 on a new handbook. An export whose fingerprint matches the last version opened or exported keeps that version number; one that differs takes the next number, with a record of the previous version's fingerprint. A file that differs only in its stored fingerprint counts as unchanged. Removing a label that was in the last version opened or exported retires it: it stays in the file with its id, outside the fingerprint, and the id is never reused. Removing a label added since deletes it.
A rubric import maps each active criterion to a label. Names, definitions, statements, must-pass flags, scales and the first pass and fail examples are copied with an origin marker so that the page shows which fields came from the rubric. A criterion now retired in the rubric adds no label, and a label built from it is retired. A re-import refreshes rubric-owned fields; hand-added labels, the rules under Counts as a fail when and Does not count when, second examples, precedence rules and quiz cases stay.
A log merge adds every incoming entry whose id is new. An incoming entry whose id already exists with identical content is skipped. An incoming entry whose id already exists with different content is renumbered, starting from the highest id in either log plus one. Internal references inside the incoming log follow the renumbering. A merge that would pass 150 entries is refused whole.
Questions
Should I use pass or fail per criterion, or one label per output?
Pass or fail per criterion is for outputs that can fail in more than one way at once, where every failing criterion is recorded and the precedence order names the primary failure. One label per output is for classification tasks where each output gets exactly one label. A rubric from the criteria builder opens as pass or fail per criterion, one label per criterion.
What does precedence do?
Under one label per output, when several labels fit, the one that comes first in the precedence order is recorded; under pass or fail per criterion, every failure is recorded and the failing criterion that comes first is the primary failure. Each rule puts one label before another, and rules chain, so A before B and B before C puts A before C. Pairs that no chain of rules connects have no order, and labelers escalate cases where both apply.
When should a labeler enter the cannot-judge token?
A labeler enters the token when the output and the handbook do not let them reach a verdict: the output is truncated, the context it depends on is missing, or a situation in the handbook's cannot-judge list applies. Questions about what a rule means go to the escalation contact. The token records no verdict, and the How it counts setting decides whether the case is left out of agreement and pass rates or sent to the escalation contact.
Why do quiz cases stay out of the handbook?
The handbook goes to labelers before they start; if it contained the quiz cases, labelers could study the answers before taking the quiz. The quiz CSV carries the cases; the answer key CSV lists each case's answer and the rule it tests, and the JSON file the author keeps carries the cases with their answers. The handbook Markdown and print carry none.
Why does this page set no agreement threshold?
Published bands for kappa disagree: the same 0.65 reads substantial on one scale and moderate on another. Our analysis of kappa thresholds for model judges found that the five published conventions are not interchangeable, so the calibration section takes the number your team sets, and the agreement calculators compute the agreement you compare against it.
Why a separate precedent log, and what is folding?
The log keeps rulings made during a labeling round: an edge case, the question, who ruled and the answer. Rulings accrue between handbook versions and apply until superseded. When a ruling has been absorbed into the handbook's rules, it is folded into the next handbook version: the log records which version absorbed it, and the ruling stays readable.
What happens when the rubric changes?
Re-import the rubric. Labels whose criteria are still active keep their hand-added rules, examples and quiz cases; labels whose criteria are now retired are marked retired in the next handbook version. The version and change record track what moved.
How does this page relate to the agreement calculators and the judge-disagreement page?
The agreement calculators compute agreement between labelers. The judge-disagreement page compares a judge's verdicts against human labels. On this page you write the rules, examples and quiz those labelers use. The page sets no agreement threshold and computes no agreement number.
Is anything I type or open uploaded?
No. What you type and any file you open are processed in this browser tab, and you can check by going offline after the page loads. This page never uploads them. The exports download to your device, and the quiz and answer key are local CSV files.