TOOL | Evaluation design
Criteria and rubric builder for LLM eval grading
Write criteria for grading one output: must and must-not statements, a pass and a fail example, and an optional scale. Checks flag vague wording, overlap and missing parts. Exports are a versioned JSON file, a Markdown file for labelers and a printable rater sheet. Set the bar a whole test set must clear on the acceptance criteria builder.
You can also open the failure list the error-analysis workbench exports, and the labeling handbook builder reads this page's JSON file.
The criteria, statements and examples below come from a fictional returns desk for an online outdoor-gear shop.
Load a failure list from the error-analysis workbench (a failure taxonomy) or a saved rubric (a grading rubric). Both are JSON files.
JSON. Drag one here, or use the box below.
Your file or pasted text is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.
Check this value.
Loading this rubric replaces the current one. Export your work first, or choose Replace to continue.
Load the returns-desk example, open a failure list or a saved rubric, or add your first criterion.
Retired criteria
Checks
Every active criterion passed these checks. They cover wording and missing parts; whether the criteria fit your task is a call for the people who own them.
How this is calculated
These checks run the same closed word lists as the rubric linter on the judge-disagreement page: a quality-word list for wording with no observable anchor, a conjunction pattern for double-barreled statements, a surface-feature list for format or length cues, a negative-opening pattern for the must and must-not lists, and a model and vendor list. The model and vendor list matches only a name written with a capital letter, so an ordinary lowercase word that shares a name's spelling does not fire; the quality-word list matches on whole words; the observable-verb list matches after the same stemming the overlap check uses. An evaluative adjective clears the check when a number, a comparison such as "at least", or a verb from the observable-verb list sits in the same statement; otherwise the check flags the statement. Two criteria overlap when the Dice coefficient of their content-word sets reaches 0.5 or higher. Content words are the name, definition and statements joined, lowercased, split on non-letters, with stopwords dropped and a trailing "ly" and then a trailing "s" stripped where at least three letters remain. The Dice coefficient is twice the shared content words divided by the two criteria's content-word counts added together.
A criterion is complete when it has at least one must or must-not statement, a pass example, a fail example, and, for a scale, every anchor and a pass line.
The overall verdict counts all active criteria when the rule is "fails on any criterion," and only the must-pass criteria when the rule is "fails only on must-pass criteria." The output fails when any counted criterion fails. When no counted criterion fails but at least one is "cannot judge," the verdict is blank. Otherwise the output passes.
A scale criterion passes when its score is at or above the pass line.
The fingerprint is a SHA-256 hash of the canonical JSON of the graded unit, overall rule, cannot-judge rule and active criteria (sorted by id), each reduced to its id, name, definition, statements, examples, scoring kind, scale and must-pass flag, with each statement list sorted, so reordering statements leaves it unchanged. The page shows its first 12 hex characters and the JSON file keeps all 64. The rubric name, check type, failure-list counts and retired criteria are left out, so the fingerprint tracks the wording grading rests on. Changing a statement changes the fingerprint, and renaming the rubric leaves it unchanged.
When a failure list is opened, active categories become criteria in count order (highest first); ties break by label. Each criterion's id is made from the label, and the source keeps the category's original id so each criterion can be traced back to its category. An imported category matches an existing criterion first by the category's original id, then by name after ignoring case and outer spaces; a match refreshes counts, and a category that matches neither way adds a new criterion. Merged categories are skipped. Opening the same list again refreshes the counts and fills a blank definition. Past 20 criteria the lowest counts are left out.
A new rubric exports as version 1. Each later export compares its fingerprint with that of the last version exported or opened in this tab: a match keeps that version, and a difference writes the next version, records what was added, retired and changed, and names the version it came from. An opened file whose stored fingerprint was edited by hand counts as unchanged. Removing a criterion that exists in the last exported or opened version retires it (kept in the file, out of the fingerprint); removing one added since deletes it.
Questions
Should I use pass or fail, or a scale?
A criterion that draws a line (the answer is right or wrong, the step was taken or not) fits pass or fail. Use a scale only when you can describe a distinct observed behavior at each point; anchors that blur together leave labelers free to pick different points for the same output. The scale offers 3, 4 or 5 points.
How is this page different from the acceptance criteria builder?
On this page you write the criteria for grading one output: what must be true, what must not be true, a pass example and a fail example. The acceptance criteria builder records the pass-rate bar a whole test set must clear, who set it and the sample size behind it, and lets you freeze them before results arrive. The rubric says whether one answer is right; the acceptance criteria say whether the feature is good enough.
Why one pass example and one fail example?
One example of each kind is the minimum that shows a labeler both sides of the line. A pass example alone leaves "fail" undefined; a fail example alone leaves "pass" to imagination. The checks on this page flag a criterion missing either one.
What do the checks catch, and what does a clean result mean?
Seven wording checks flag a quality word with no anchor, two things at once, overlapping criteria, a surface proxy, a negative opening, a double negative, and a model name. Eight completeness checks flag missing statements, examples, anchors, a cannot-judge rule, a must-pass criterion, a scale pass line, and anchors that read the same. A clean result means every criterion has its statements, examples and anchors and no wording check fired. Whether the criteria fit your task is a call for the people who own them.
What happens when I change criteria after people have labeled outputs?
The Markdown, JSON and rater sheet carry a version number and a fingerprint. Each export compares the grading wording with the last version exported or opened in this tab: when it differs, the export carries the next version and lists what was added, retired and changed. The page keeps no copy, so after a reload, open the last exported JSON before you edit. A retired criterion stays in the file with its id, so result rows keyed to it can still be traced. Relabeling decisions are yours to make.
How does the "must-pass" flag work with the overall verdict?
Under "fails only on must-pass criteria," only a fail on a flagged criterion fails the overall verdict. Under "fails on any criterion," the flag has no effect. When no counted criterion fails but at least one is "cannot judge," the verdict is blank: a blank verdict is neither pass nor fail.
How do a harness and a model judge use the exported file?
The criterion id becomes the grader name in result rows (the scorer column). Each grader writes pass or fail in pass and, for a scale, the point in score. A "cannot judge" verdict is a blank pass, with the agreed cannot-judge word in the raw score (the score exactly as the grader wrote it). The overall verdict goes on a row with a blank scorer, by the rubric's rule: under "fails on any criterion," any fail means fail; under "fails only on must-pass criteria," only a must-pass fail does.
I have error-analysis notes. What should I open here?
Open the failure list from your review. Active categories become criteria, ordered by count; each criterion's id is made from the category's label, and the category's original id is kept as the source. Counts from a non-random review order rank failures by frequency; a rate needs a random sample. A session file is not read here. This page reads a failure list (a failure taxonomy) that the error-analysis workbench exports with its Taxonomy (JSON) button. Export it from the workbench first.
Is my rubric uploaded anywhere?
No. Your rubric is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it. The Markdown and JSON exports are files your browser downloads; the rater sheet prints and the PRD copy goes to your clipboard.