TOOL | Eval statistics
Import eval results from promptfoo, Inspect, lm-eval or OpenAI
Paste rows in the six result fields, or load a file from promptfoo, Inspect AI, lm-eval or OpenAI Evals. The importer detects the source and maps each record to per-case rows you can download, keeping the grader scores and error states the source wrote. Errored attempts keep no verdict and are never dropped. The pass-rate calculator's unscored-runs mode takes one grader's counts on one version, errored ones included, and shows the rate under three conventions.
These settings were loaded from the link you opened.
Paste rows in the six result fields (case, version, repeat, grader, score, pass or fail) or load a results file from promptfoo, Inspect, lm-eval or OpenAI. Load the returns-desk example to see what comes back.
The counts and rows below come from the returns-desk example.
Counts by version and grader
| Version | Grader | Scale | Rows | Pass | Fail | No verdict | Of which errored | By kind |
|---|
| item | system | repeat | scorer | score | pass | error |
|---|
Each row carries one grader's score on one case, one version and one repeat. A row from an attempt that ended in an error carries no verdict.
Enter the pass and fail counts from one row of the table above in the pass-rate calculator's unscored-runs mode, and that row's errored count in the unscored box that fits, such as Timed out; the mode shows the rate under three conventions.
How this is calculated
The importer detects the source, maps each record to per-case rows, and checks every row against the six-field result format.
Six-field rows. Six-field rows are read as they are. A case and a version are required; a blank repeat reads as 1, a blank grader as the overall verdict, and a blank score or pass as no score or no verdict.
promptfoo (results file as of version 0.124.1). Each entry in the file's results list becomes one attempt. The case id comes from a named field in vars when the case id field is set, from the description when it uniquely identifies one set of variables, or from a stable hash of the variables. The version name comes from the provider and prompt labels. Each grader's result produces one row. One overall row records the result's own success flag. Each named score that no grader already names adds one row with no verdict. Errored results (failure reason 2) keep no verdict; the error kind is not stated by promptfoo. A cross-check compares the file's success and failure counts, and its error count when the file states one, against the overall verdict rows. Without a stated error count, the file's failures are compared with fail and errored rows together.
Inspect AI (log schema 2, as of version 0.3.281). An .eval file is a ZIP archive. The importer reads only the members the archive's directory lists, checks each one's checksum and declared size, and reads stored, deflate and zstd members. The case id is the sample's id, and the epoch is the repeat. Grader letters map to verdicts: C passes, I and N fail, and P carries no verdict. No letter becomes a number; the letter is kept exactly as the log wrote it. A sample that ended in error gets one row per grader, each with no verdict. A sample halted by a limit keeps its grader verdict. Tokens are summed over the model usage fields, and cost is read from the sample's total cost when the log records one. The cross-check compares the log's declared totals against the samples found.
Language Model Evaluation Harness (lm-eval, --log_samples output as of version 0.4.13). Each line in the samples JSONL is one document and one filter. The case id comes from a named field in doc when the case id field is set, or from doc_hash by default. The version name is typed by the reader, because the file does not carry it. A metric carries a verdict only when every value of that metric in the file is exactly 0 or 1, and then 1 passes and 0 fails; any other value leaves that metric's verdicts blank. A bare NaN metric becomes a no-score row. Duplicate documents for one filter are numbered as repeats.
OpenAI Evals (output items, API specification 2.3.0 at commit 6ca9cb2b, read 2026-10-17). Output items are the primary source. The case id comes from a named field in the dataset item when the case id field is set, or from a stable key derived from the item. Each results entry produces one row; a trailing criterion UUID suffix is stripped from the grader name when the stripped names stay unique, and the full id is kept with the row. The overall row comes from the item's status. The dashboard export (eval_items_*.jsonl) is read with a provisional label: its format has no published specification.
Inspect logs, lm-eval samples and the dashboard export write no overall verdict row, and none is built from the graders' verdicts.
Repeats. Every repeat is stored counted from 1 (the six-field format's convention). A source that counts from 0 is shifted and the original base is recorded. When no explicit repeat field exists, occurrences are numbered by file order.
Errored rows. An attempt that ended in error gets no verdict and no score. The error message is kept, and so is the error kind (infrastructure, the model or the grader) where the source says which; otherwise the kind is left blank. Consuming tools apply their own convention for how to count errored rows.
Scales. A score is never rescaled. A scale that a six-field JSON document declares is kept as declared. Otherwise a grader's scale is 0 to 1 (rate, in the JSON document) only where the source fixes it, as Inspect's true-or-false scores and 0-or-1 lm-eval metrics do; every other grader's scale is open, meaning no range is assumed.
Show the detection order
Detection. The source is matched by structure in this order. Bytes that start with the ZIP signature are an archive; with Inspect-specific members, the source is an Inspect .eval log, otherwise a ZIP with no recognized content. A JSON document is classified by its top-level keys: a promptfoo results object, an Inspect JSON log, an OpenAI output items response (a list page, an array or a single item), an OpenAI object that is not output items, an lm-eval aggregate file, rows in the six result fields, or test cases with no results. JSON lines are classified by the first line that parses: lm-eval samples, OpenAI output items, the OpenAI dashboard export, promptfoo JSONL, test cases with no results, or rows in the six result fields. A CSV or TSV with a bracketed provider header is a promptfoo table export; any other CSV or TSV is read as six-field rows. Everything else is unsupported.
Worked example
The returns-desk example is a promptfoo results file with 80 results: 40 cases, each answered by two versions, returns-assistant-v1 and returns-assistant-v2, and graded by label_match and policy_judge. Each result becomes three rows, one per grader plus its overall verdict, so the import writes 240 rows. On v1, label_match passes 36 of 40, policy_judge 30 of 40 and the overall verdict 28 of 40; on v2, the same three read 37, 33 and 31 of 40. The file's own summary reports 59 successes, 21 failures and 0 errors, which matches 28 plus 31 overall passes and 12 plus 9 overall fails. Case rd-002 on v1 shows the mapping: label_match scores 1 and passes, policy_judge scores 0 and fails, and the overall row scores 0.5 and fails.
Questions
Why do errored cases have no verdict?
An attempt that ended in an error, such as a timeout, an infrastructure fault or a grader failure, has no grader score to record. Recording it as a fail picks one convention for you, and dropping it hides it from every count. The importer keeps the row with no verdict so that you can see what happened and choose how it enters a rate. The pass-rate calculator's unscored-runs mode shows the rate three ways: scored only, all attempted (errors counted as failures), and a bound pair whose upper edge counts them as passes.
Why does the Inspect log show a different accuracy than these counts?
Inspect's summary scores are computed over completed samples only. When some samples ended in error, such as a timeout or an infrastructure fault, the log's own figures leave them out. In the Inspect example with timed-out cases, 3 of 40 cases timed out, so the log reports label_match as 34 of 37 completed samples. The importer keeps all 40 cases for each grader, with the 3 timed-out ones carrying no verdict, so the table shows label_match at 34 passes, 3 fails and 3 with no verdict.
Why do lm-eval samples need a version name?
lm-eval samples files carry metric scores per document, but the file itself does not name the model that produced them. A comparison pairs rows by case and version, and without a version label the rows cannot be paired. The same applies to the OpenAI dashboard export. Type a version name, such as the model or prompt you ran, in the version name field before importing.
How are repeats numbered?
Every repeat is counted from 1. When the source carries an explicit repeat number (Inspect's epoch, or an older promptfoo repeat index counted from 0), the importer uses it, shifting a 0-based source up by one. When no repeat field exists, the importer numbers occurrences of the same case and version in file order. Either way, repeat 1 is the first attempt at that case.
Why is the OpenAI dashboard export marked provisional?
The eval_items_*.jsonl export from the OpenAI dashboard has no published specification. The importer reads it with a mapping taken from one existing reader of the format, so the mapping may change if the format changes. The API's output items have a published specification and are the primary source.
What do the Inspect grader letters C, I, P and N become?
C (correct) becomes pass true. I (incorrect) and N (noanswer) become pass false. P (partial) carries no verdict. No letter becomes a number: the score is left blank, and the download keeps the letter exactly as the log wrote it. A true or false score becomes 1 or 0 with the same verdict, and a numeric score is kept as written with no verdict.
Which promptfoo exports work, and how do I shrink one under 10 MB?
The JSON output and the JSONL output keep per-grader scores and per-case variables. The CSV table export rounds scores and drops the per-grader breakdown, so this importer does not read it. If a JSON file is too large, switch to JSONL output, which the importer also reads, or set the environment variables PROMPTFOO_STRIP_PROMPT_TEXT and PROMPTFOO_STRIP_RESPONSE_OUTPUT to strip the prompt text and the model output.
How do I make case ids match across harnesses?
If every source's cases carry a shared key, such as case_id, type it in the case id field under More options: the importer reads it from promptfoo's vars, lm-eval's doc and the OpenAI item's datasource_item, and uses its value as the case id. Inspect ignores the field, so to match an Inspect log, pick a key whose values equal its sample ids. Without it, each source uses its own default: promptfoo uses the test description, or a hash of the variables when a description is missing or shared by different variables; lm-eval uses doc_hash; OpenAI uses a key from the dataset item; Inspect always uses the sample id.
Is my file uploaded anywhere?
Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it. The "Load the returns-desk example" button fetches a demo file from this site.