TOOL | Eval statistics
Eval results auditor: find errored, empty and unreadable rows
Paste rows or load a results file, and the page classifies every row that ended without a verdict: errored attempts, NaN scores, empty rows, and unparsed grades. It reports the pass rate three ways. Scored only leaves unscored rows out. All attempted counts them as failures. The upper edge counts them as passes. Our review of silent-failure studies traced one paper's headline rate to the attempts that finished, the scored-only reading. For typed counts, use the Unscored runs mode.
Paste result rows or load a file. Each row carries a case, a version, a repeat, a grader, a score, and pass or fail. The page also reads promptfoo, Inspect, lm-evaluation-harness and OpenAI results files. Load the returns-desk example to see the audit.
How each harness records unscored rows
Inspect (0.3.281)
An Inspect log marks an unscored sample with a NaN score, which aggregate metrics and reducers skip. Its value_to_float function maps C to 1, P to 0.5, and I and N to 0; casts numbers, booleans, numeric strings and the words "yes", "no", "true" and "false"; and returns 0 with a warning for anything else. Logs carry total_samples and completed_samples, and can carry scored_samples and unscored_samples per scorer. Checked 2026-10-17 against Inspect 0.3.281 source.
OpenAI Evals API (openapi 2.3.0 at 6ca9cb2b)
Result counts separate errored, failed, and passed items. A score_model grader defaults non-numeric model output to a score of 0. A python grader returns a score of 0 when the script raises an exception. List pages return 20 items by default; a has_more flag signals more pages. The status filter accepts only pass or fail, so a filtered export omits errored items. Checked 2026-10-17 against openapi.json at commit 6ca9cb2b and OpenAI's graders guide.
promptfoo (0.124.1)
The success field is false on a failed assertion and on a provider error. The failureReason field separates them: 1 for an assertion failure, 2 for an error. Stats count successes, failures and errors as separate totals. Checked 2026-10-17 against promptfoo 0.124.1 source.
lm-evaluation-harness (0.4.13)
NaN is written as a bare token. Repeats of one document sit inside one row. Each document gets one row per filter, so a task with two filters writes every document twice. Checked 2026-10-17 against lm-evaluation-harness 0.4.13 source.
How this is calculated
Each row is classified into exactly one class by the first matching rule, then grouped by version and grader. Within each group, the page computes the scored-only rate, the all-attempted rate, the upper edge, and the envelope between all attempted and the upper edge, each with a two-sided Wilson score interval at the chosen confidence level.
Classification. Duplicate and unattributable rows are refused. The remaining rows are classified in priority order: NaN score, unparsed grade, other unreadable row, errored, completed (pass or fail), graded with no verdict, no score and no verdict and no error. Each row enters exactly one class; errored rows carry their error kind (infrastructure, model, grader, or unstated).
The three conventions. Scored only divides passes by scored rows, leaving out every unscored row. All attempted divides passes by all attempted rows, counting every unscored row as a failure. The upper edge divides passes plus all unscored rows by all attempted, counting every unscored row as a pass. When there are no unscored rows, the three conventions give the same number and the page reports one rate.
The envelope. The lower bound of the all-attempted interval to the upper bound of the upper-edge interval is the range the true rate plausibly sits in when you do not know how unscored rows would have turned out. The wider the gap between scored only and all attempted, the more the result depends on that assumption.
Declared-figure match. When the file carries its own reported figure, the page checks whether it equals the scored-only rate, the all-attempted rate, or the rate with unparsed grades counted as failures. A match names the convention the file used. When none matches, the candidates are shown with their values.
The zeroing check. Three paths turn a grade that could not be read into a score of 0: OpenAI's score_model grader on non-numeric output, OpenAI's python grader on an exception, and Inspect's value_to_float function on any value it does not recognize. When every passing row for a grader carries a recognized raw grade, each failing row whose raw grade is unrecognized text is reclassified from completed to unparsed grade. The check does not run when passing rows themselves carry free text, or when no raw grade is present.
Unequal repeats and the each-case-once rate. When cases in a group have different numbers of repeats, the pooled rate counts every row equally, so cases with more repeats contribute more weight. The each-case-once rate averages each case's own scored-only rate (cases with no scored rows excluded), giving each case equal weight.
Wilson two-sided. Each group is computed independently: the page reads each version on its own, so no paired test applies and no difference between versions is tested. Each interval covers one group, with no multiplicity correction across the groups shown.
Questions
Which number should I report?
Report the scored-only rate with its scored and unscored counts beside it, or report the envelope. Scored only is a fair estimate when the unscored rows are like the rest, such as timeouts that struck cases at random. When unscored rows are model errors, such as a content filter or an empty reply, scored only leaves real failures out. For a release bar, use all attempted: it counts every unscored row as a failure, so the bar cannot be met by dropping the hard cases.
Why does an errored attempt count as a failure in the all-attempted convention?
The all-attempted convention answers "what is the worst this rate could be?" An attempt that errored never demonstrated a correct answer, so counting it as a pass would assume the best case. The upper-edge convention makes that assumption explicitly, and the envelope between them shows how much the answer depends on it. If infrastructure errors are the cause, rerunning those cases and replacing the errored rows with the new results narrows the envelope.
What counts as unscored here?
A row is unscored when it reached the page without a usable pass or fail verdict, or when it carries an error. That includes errored rows (timeouts, rate limits, content filters, grader failures), NaN scores, grades this page could not parse, other unreadable rows (a blank case id, a score off its scale), rows with no score, no verdict and no error, and, on a grader that gives verdicts, rows with a score but no verdict. Duplicates and rows with no version are refused and do not count as attempts.
My harness printed a different number. Why?
The harness may have dropped errored, timed-out, or empty attempts before computing the rate, which gives you the scored-only number on a smaller denominator. Or it may have defaulted an unrecognizable grade to 0, counted that row as a failure, and divided by the rows it kept. The declared-figure match shows which convention produced the number your file reports. The harness notes, folded under the results, list each harness's behavior at the version this page was checked against.
Why are NaN rows refused and still counted?
A NaN in the score column is not a number the page can use, so the row is refused by the parser. It still counts as an attempted row because the case was submitted and a result came back. Leaving it out would shrink the count of attempts and hide the gap that the all-attempted rate shows.
What is the zeroing pattern?
Three paths turn a grade that could not be read into a score of 0: OpenAI's score_model grader on non-numeric output, OpenAI's python grader on an exception, and Inspect's value_to_float function on any value it does not recognize. When every passing row for a grader carries a recognized raw grade, this page reclassifies each failing row with unrecognized raw text from completed to unparsed grade. The file's own rate counted those rows as failures; this page shows them apart from real zeros.
What happens when cases have different numbers of repeats?
The pooled rate divides total passes by total scored rows, so a case with five repeats contributes five times the weight of a case with one. The each-case-once rate averages each case's own scored-only rate, giving each case equal weight. The two can diverge when cases with more repeats pass at a different rate than cases with fewer. The pooled rate is what a harness that flattens all rows reports.
How does this differ from the Unscored runs mode?
The Unscored runs mode on the pass rate confidence interval page takes typed counts of passes, fails and each kind of unscored attempt, and shows the same three conventions and the envelope, with a Clopper-Pearson column beside Wilson. This page reads a file, classifies rows from what the file recorded, runs the checks against the file's own declared figures, and shows harness-specific notes. Use the Unscored runs mode when you already know the counts; use this page when you have a file and want to see what is in it.
Is my file uploaded anywhere?
No. Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it. The "Load the returns-desk example" button fetches a demo file from this site.