TOOL | Evaluation design
Error analysis workbench for LLM traces and eval results
Load a file of traces or eval results. Read each case, mark pass or fail, write a critique, tag a failure code and note the first wrong step. Codes build an adaptive failure taxonomy; a Pareto chart shows what fails most. The step draws on the trajectory failure analysis. Tagged rows and the taxonomy export. Your file stays in this browser; a session file is the save. On a larger file, sample first on the review sample drawer.
These settings came from the link you opened. Your file was not in it.
CSV, TSV, JSONL or JSON. The file is read once and held in memory.
CSV, TSV, JSONL, NDJSON, or JSON. Drag one here, or use the box below.
Paste rows in any supported format.
Check this value.
Up to 10 MB and 200,000 rows. On a larger file, draw a random sample first.
Your file or pasted text is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.
Resume from a session this page exported. The session holds your notes and case ids, so load the file it was made from as well.
Case review
A short note on what went wrong or what was correct.
A short label for this type of failure. Codes with the same text merge into one category.
A code can be at most 60 characters.
The first step in this case that went wrong. Later errors often follow from it.
Shortcuts
- p
- Pass
- f
- Fail
- u
- Unsure
- j or Right
- Next
- k or Left
- Previous
- g
- Next unreviewed
- c
- Critique
- t
- Code
- Escape
- Leave field
- ?
- Shortcuts
This review uses the example, a fictional returns desk for an online outdoor-gear shop. The figures below fill in as you mark verdicts and codes.
Review summary
The failure share is the proportion of judged cases a reviewer marked as failing, and the taxonomy counts how often each type of failure appeared, so that you can see which ones occur most often.
Turn each frequent category into a pass or fail criterion for the next release, and keep the tagged rows as human labels for checking a model judge against your reviewers.
Failure share by slice, each with its own two-sided 95% Wilson score interval. The table does not compare slices with each other and applies no correction for comparing many slices.
| Slice | Judged | Failing | Share | 95% interval |
|---|
| Category | Code count | Case count | Share of coded failures | Rate | 95% interval |
|---|
Another category already has this name. Choose a different name.
Failure categories by count
| Category | Count | Share of coded failures | Cumulative |
|---|
Distinct failure categories as coded failures accumulate
| Coded failures | Distinct categories |
|---|
Load the example, pick or paste a file of traces or eval results, or resume from a session file. Your file stays in this browser and is never uploaded.
No working copy is stored in this browser. The session file is the only save.
failure share = failing cases / (passing + failing cases), with a two-sided Wilson score interval at 95%How?
How this is calculated
The queue holds every parseable case from the file (in failures-only order, only the cases the file marks as failing or errored), in the order the review-order setting chose. Random order is a seeded shuffle with the Mulberry32 generator, the same generator the review sample drawer uses. The seed and file together fix the order.
The counted set is the prefix of consecutive reviewed cases from position 1 in random order. In file order the counted set exists only once every case has a verdict (a census). In failures-only order no counted set exists, so rates are hidden and only counts are shown. Cases reviewed past a gap in random order count toward totals and the taxonomy but not toward rates.
Each case gets a verdict (pass, fail, or unsure), an optional critique, and on a failing case an optional code and an optional first failing step. Unsure cases are left out of every denominator.
The failure share is F/J, where F is the failing count and J is the passing-plus-failing count over the counted set. The interval is a two-sided Wilson score interval at 95%. Category rates are each category's count within the counted set divided by J, a lower bound while uncoded failures remain.
Each failing case takes one code. A new code creates a category. Merging one category into another moves its codes; after a chain of merges, every old category points straight to the one that absorbed it. The Pareto chart sorts active categories by count, descending, with ties broken by label. The saturation chart tracks distinct active categories as coded failures accumulate in queue order; the last-20 window counts new categories among the last 20 coded failures in queue order.
The session file is a JSON document that stores the queue order, every annotation, and the full taxonomy. Its cap is 16 MB. The session records the source file's SHA-256 hash so that the page can tell whether the same file or a different one is loaded.
Why random order?
Any prefix of a random queue is a simple random sample. Stop reviewing at any point and the failure share so far has a known interval. File order gives a census once every case is reviewed, but nothing in between. Random order makes a partial review useful.
Show the formula
Failure share F/J: the number of cases reviewed as failing divided by the number reviewed as passing or failing, with unsure cases excluded. The interval is a two-sided Wilson score interval at 95%. Category rate c/J: the count of cases coded with this category divided by the same denominator. While uncoded failures remain, every category rate is a lower bound.
Worked example
Suppose a reviewer works through all 40 traces in the returns-desk example (seed 1) and marks 30 as passing, 10 as failing and none as unsure. Failure share 10/40 = 25.0% [14.2, 40.2]. Their ten codes start as ten categories, and merging similar ones leaves five: Wrong return action (4, rate 10.0% [4.0, 23.1]), Missed escalation (2, 5.0% [1.4, 16.5]), Unsupported claim (2, 5.0%), Gave up on lookup (1, 2.5% [0.4, 12.9]), Wrong language (1, 2.5%). The Pareto chart shows 40% of coded failures in the largest category; the table reads 60% cumulative after two and 80% after three. The saturation curve reaches 5 distinct categories by 10 coded failures.
Formula: failure share = failing cases / (passing + failing cases), with a two-sided Wilson score interval at 95%
Questions
Where are my file and my notes stored?
Your file is held in this browser tab while the page is open. Your notes are held there too and written to the session file you download, which holds each case's id but no transcript or other row text. The page never uploads your file or your notes. If you close the tab, changes since your last session file are gone, unless you turned on the working copy in this browser. Safari erases that copy after 7 days of Safari use without a visit to this page.
Why a session file?
A session file is yours to keep, move, and share. It holds your verdicts, critiques, codes and categories, the queue order, the column mapping, and each case's id from your file. It holds no transcript or other row text. Loading it into this page with the same source file resumes exactly where you left off. Loading it with a different file attaches notes by case id where the ids match.
When should I stop reviewing?
There is no settled rule. The saturation curve shows whether new failure categories are still appearing. If the last-20 window shows no new category, each of the last 20 coded failures fell into a category you already had. The interval on the failure share shows how precise the estimate is at the current count, and reviewing more cases narrows it. Both are facts to weigh against the time a review takes; the page does not set a threshold.
How is this different from the agent trace analyzer?
The agent trace analyzer counts steps, repeated tool calls and handoffs in a trace file by itself, from the file's structure. On this page a person reads each case, marks pass or fail, and names each failure, so that the counts at the end are counts of failures someone judged.
Why note the first failing step?
When an early step goes wrong, later steps often follow from it, so the first failing step points a fix at where the failure started. The tagged rows carry it on each failing case where you marked it, so that you can count it across cases. The workbench records both the position shown on the page and the index in your file, so that it joins back to the source file.
How do codes become categories?
Each failing case gets one code, a short text label. Codes with the same text (case-insensitive) share a category. Start with the first word that fits; once you have seen enough failures, merge similar categories. Merging moves codes and keeps history. The taxonomy export lists every active category, its codes, and a count.
Why are rates hidden in file order and failures-only order?
In file order the denominator exists only once every case has been reviewed, because the order is not random and a partial review could be biased by whatever the file's ordering favors. In failures-only order the denominator never covers the full population. Counts and shares of coded failures are shown in every order.
What file formats does the workbench read?
CSV, TSV, JSONL (one JSON object per line) and JSON (an array of objects). The auto detector tries each shape in order: step rows (every row has traceId and stepIndex), result rows (columns match item and system), conversations (any row has an array of objects with a string role), then a plain table. In a result file, each output (one case, one version, one repeat) is reviewed once, with every grader's row shown together. Column mapping lets you point each field at a column or a one-level nested path in your file.