Error analysis and labeling tools
From a pile of traces to a labeled set: review cases, build a failure-coding taxonomy, set the labeling criteria, and write a handbook for the people who label.
How the work proceeds
Error analysis is the step between “something went wrong” and “we can measure it.” You review cases one at a time, coding each failure into a named category. Those categories drive everything else. For a large file, draw a fair sample before reviewing, because going through every row takes too long and a biased pick hides the prevalence. Labeling criteria turn the named categories into pass-or-fail rules with examples. A handbook gives those criteria to the people who label, with a quiz and a log of the edge calls that come up. The pages below read your files in the browser and upload nothing.
| What you have | The instrument |
|---|---|
| Traces or eval results and no error taxonomy yet | review cases one at a time and build a failure taxonomy |
| A large file to review, with no way to pick a fair sample | draw a random or stratified review sample |
| Failure categories named, with no written criteria for pass or fail | turn the categories into criteria with examples |
| Criteria ready, with no handbook for the people who will label | build the handbook, quiz and precedent log |
How coding agents fail, a trajectory study of 1,794 runs, found the decisive error at a median of step 7, so a case review starts early in the trace rather than at the end. AdaMAST induces failure categories from an agent system’s own traces, giving the vocabulary the review step needs. The labeling half draws on Is your LLM-as-a-judge reliable?, which measures how closely an LLM judge tracks human labels.
Readers often ask whether to review cases first or write criteria first. Review cases first if the team has no shared vocabulary for what goes wrong. The categories that emerge from a review sample are usually different from the ones the team assumed, and a rubric written before seeing the data locks in the wrong distinctions. Write criteria first if the team already agrees on what “pass” means.
Tools in this topic
Error analysis workbench for LLM traces and eval results
Review traces or eval results one case at a time: mark pass or fail, code each failure, note its first failing step, and build a failure taxonomy. Your work saves to a session file you download.
Tool | Error analysis and labelingReview sample drawer: random and stratified sampling for eval review
Take a reproducible random, stratified or fixed-rate review sample from a large CSV or JSONL, with each row's inclusion probability and a manifest of the settings. Your file stays in this browser tab.
Tool | Error analysis and labelingCriteria and rubric builder for LLM eval grading
Turn failure categories into pass or fail criteria with must and must-not statements and examples, lint vague wording, and version the rubric as a file for labelers and whoever builds the checks.
Tool | Error analysis and labelingBuild a labeling handbook from criteria, with rules, examples and a quiz
Turn criteria into a labeling handbook with rules, examples, precedence and a cannot-judge policy, plus a quiz and a precedent log. The page exports the handbook, quiz, answer key and log as files.