TOOL | Evaluation design
Score a results file with code graders
Load or paste a results file, pick code graders, and get pass, fail and near-miss counts for each grader, plus a scored file. For each failing row, the near-miss probe names the formatting step that would make it pass, so that you can separate wrong answers from formatting gaps before you report a pass rate. Ten graders run in this browser tab, and your file stays here. To try one pair first, use the grader playground.
The figures below use the returns-desk example: 40 cases, version returns-assistant-v2, graded by exact
match and normalized exact match.
Pick a file, paste rows, or load the example with one click.
| Grader | Graded | Passed | Failed | Not judged | Errors | Near misses | Mean score |
|---|
- Corpus BLEU
- Mean sentence BLEU
- Corpus chrF
- Mean sentence chrF
| Case | Grader | Step that would pass it |
|---|
Each grader is counted on its own. A row that fails exact match and passes normalized exact match differs only in what the normalization removes, such as case or surrounding whitespace.
The scored file gives counts. For the interval around a pass rate, take the count to the pass rate calculator. To try a single pair, use the grader playground.
The scored file has one row per case, version, repeat and grader, in the six-field result format this site's statistics pages read.
Each grader scores every row independently, then counts and near misses are tallied.How?
How this is calculated
The page reads CSV, TSV, JSON lines, JSON arrays and JSON documents with a rows array
through a shared parser, up to 10 MB and 200,000 rows. It maps columns by name: id or
case_id for the case, expected or reference for the reference
answer, output or actual for the model's response, and others from a
suggestion list you can override. A row's key is its combination of id, version and repeat. A second
row with the same key is rejected, and the first row's position is named.
Each grader runs the same logic as the grader playground. Exact match, normalized match, contains, regex, numeric, JSON Schema, token F1, BLEU, ROUGE-L and chrF produce one scored row per case per grader. The near-miss probe runs on every failing row and names the formatting step or mode change that would flip it.
For BLEU, the page shows both the mean of sentence BLEU scores and corpus BLEU. Corpus BLEU sums n-gram statistics across all rows, applies exponential smoothing without effective order, and computes one brevity penalty from the summed output and reference lengths (each row contributing its closest reference). The two numbers differ because sentence BLEU uses per-pair brevity penalties, smoothing and effective order, and corpus BLEU pools the counts with no effective order. Stating the configuration matters (Post, "A call for clarity in reporting BLEU scores," WMT 2018).
For chrF, the page shows both corpus chrF (summed per-order statistics with each row's best reference) and the mean of sentence chrF scores. These BLEU and chrF figures follow the settings above and have not been compared against a reference library.
The near-miss probe is the same as the grader playground's: on each fail, it tries each formatting step on its own and reports which ones close the gap. A file where most exact-match fails have a near miss at the trim or lowercase step usually needs a normalization pipeline.
Per-row assertions from the file run as extra graders on their row. Known kinds (exact,
equals, normalized, contains, icontains,
contains-any, contains-all, icontains-any,
icontains-all, starts-with, regex, is-json,
json-schema, numeric, token-f1, bleu,
rouge-l, chrf) run; unknown kinds (such as llm-rubric,
javascript, similar, rouge-n or levenshtein) are
counted by kind and listed, never counted as passes.
When the overall row is on, one additional row per case, version and repeat summarizes the six pass-or-fail graders (exact match, normalized exact match, contains, regex, numeric and JSON Schema) under the chosen rule. Token F1, BLEU, ROUGE-L, chrF and per-row assertions do not count toward the overall row, even when a threshold gives one of the four graders a pass or fail. Under "All pass", every selected grader must pass, and the row is undecided when none failed and at least one was not judged. Under "Any passes", one pass is enough, and the row is undecided when none passed and at least one was not judged. An undecided row's pass and score are blank, and the error column holds not-judged. When no pass-or-fail grader is selected, error holds no-binary-grader. Every overall row records the rule in its metadata.
Formula: Each grader scores every row independently, then counts and near misses are tallied.
Questions
What file formats and columns does this page read?
CSV, TSV, JSON lines, a JSON array of objects, or a JSON document with a rows array. The
page suggests column mappings by name (for example id, expected,
output, reference, actual), and you can change them. Object
fields one level deep appear as dotted names (for example metadata.harnessPass). The
scored file carries the six result columns item, system, repeat,
scorer, score, pass: the case id, the version, the repeat, the
grader, its score, and pass or fail, plus the output text when you include it.
What is a near miss?
A near miss is a failing row that passes after a formatting change. The probe tries seven steps one at a time: trim, lowercase, NFKC, collapse whitespace, strip a trailing period or exclamation mark, join words with underscores, and strip outer quotes. When no single step works but all seven together do, the report says "several formatting differences." A near miss can also name the case-insensitive toggle, a code fence, another number reading or a score within 0.1 below the threshold.
Why does corpus BLEU differ from the mean of sentence BLEU?
Corpus BLEU pools n-gram counts across all rows, applies one brevity penalty to the summed lengths, and uses no effective order. Sentence BLEU gives each row its own brevity penalty, smoothing and effective order, and the mean averages those scores. On short outputs the two can differ in either direction: with BLEU on, the returns-desk example gives a mean sentence BLEU of 0.846 and a corpus BLEU of 0.743. Report which one you quote, with its settings.
Why is there no single score across all graders?
Each grader checks a different property. A row that passes token F1 and fails exact match shares enough words with the reference but differs from it character by character. Combining them would hide what the difference is. If you need one verdict per case, turn on the overall row and choose whether all the selected graders (exact match, normalized exact match, contains, regex, numeric and JSON Schema) must pass, or any one is enough.
Which per-row assertion kinds does this page run?
exact and equals, normalized, contains (and
icontains, contains-any, contains-all, icontains-any,
icontains-all), starts-with, regex, is-json (routes to
json-schema when a schema is given), json-schema, numeric,
token-f1, bleu, rouge-l and chrf. Kinds this page
does not implement (such as llm-rubric, javascript, similar,
rouge-n or levenshtein) are listed with their counts and skipped.
Why were some rows left out?
A row is left out when it has no id, no output, no expected answer (if an active grader needs one), a
duplicate key (same id, version and repeat as an earlier row), a repeat that is not a whole number of
0 or more, an expected kind other than exact, any-of or label,
or malformed assertions. Any line or entry that could not be read as a row is also left out. The count tells you how many, broken down by reason, with the first twenty
row numbers.
Where is the confidence interval on the pass rate?
This page gives pass counts. For the interval, take the count (for example 34 of 40) to the pass rate calculator, which computes the Wilson interval at your chosen confidence level.
How do I reproduce this run in my harness?
Export the grader configuration. It records every grader, its parameters and the column mapping. The graders themselves are documented on the grader playground; the normalization pipeline can be exported as Python or TypeScript from the normalizer builder.
Is my file sent anywhere?
Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.