Old harness pass rate
81.00%
TOOL | Eval statistics
Separate what the harness moved from what your system moved. Feed the overlap bridge and your eval results to see the harness offset, the system-attributable change and a restated baseline.
Upload an overlap bridge and your baseline and post-migration results, with OpenAI output items accepted as the old harness's verdicts. The page separates the harness offset from the system-attributable change and puts an interval on each; with the bridge alone it returns the offset and a restated baseline. The live guide covers the shutdown and the alternatives; this tool takes your files and quantifies the difference. For the exact paired test, see the McNemar calculator.
Showing your last valid result. Update the inputs above to recompute.
These figures are the returns-desk example, a fictional returns desk for an online outdoor-gear shop. The two harnesses agree on all 40 items, and the 95% interval on the harness offset is still [-0.0585, +0.0585].
System-attributable change (pts)
+1.00
Observed change +11.00 pts, harness offset +10.00 pts, over 1,000 overlap items.
This interval is approximate because the overlap and observed datasets share items.
Old harness pass rate
81.00%
New harness pass rate
91.00%
Every delta is new minus old. The system-attributable change carries an interval and no verdict.
| Stage | Delta (pts) | Interval | Reads as |
|---|---|---|---|
| Harness offset | +10.00 | [+0.0797, +0.1213] | excludes zero |
| Observed change | +11.00 | [+0.0804, +0.1397] | excludes zero |
| System-attributable change | +1.00 | [-0.0264, +0.0460] |
Restated baseline
91.00%
Baseline as recorded
81.00%
The restated baseline falls outside 0 to 100 percent. The arithmetic value is shown as it is.
The observed change is +11.00 points (interval excludes zero). The harness offset accounts for +10.00 points (interval excludes zero). The remainder attributable to the system is +1.00 points, and the interval on that difference is approximate.
Add the dataset named above to complete the attribution.
The link carries the settings only. The eval-set file is not included.
The page runs three stages. First it measures the harness offset from the overlap run: both harnesses scored the same frozen items on the same system, so the only thing that moved between the two columns is the instrument. The overlap 2x2 is old-harness-major, the old harness indexing the rows and the new one the columns, and the interval is the Newcombe paired construction.
Second, it measures the observed change between the baseline and the post-migration results. When both datasets share an item set the page uses a second paired 2x2 on the same pattern. When they do not, it uses the two summary rates and the Newcombe independent construction.
Third, it subtracts the harness offset from the observed change. The system-attributable change is the part of the observed movement that the instrument did not cause.
Every delta is new minus old. A positive number means the rate went up after the change. The harness offset, the observed change and the system-attributable change all follow the same convention.
The interval on the system-attributable change is approximate. The two estimates share items, and the tool does not model that overlap, so the interval can read wider or narrower than the true coverage.
The interval on the system-attributable change is computed by the method of variance estimates recovery (Zou and Donner, 2008). That method assumes the two estimates are independent. The overlap run and the post-migration run can share items, so the true variance carries a covariance term this page does not model. The interval is reported as approximate rather than omitted, and it carries no verdict.
When the source is OpenAI output items, the page reads each item's verdict from its status (or from the chosen grader's result), pairs it with the matching row in the other file by the chosen id field, and applies the selected unscored convention to items that have no verdict. The joined pairs become a flat table, one row per item, and go through the same reader a pasted file does, so every interval and delta after that is computed as described above.
The interval on that difference is approximate. The harness-offset estimate and the observed-change estimate share items, so they are not independent, and the interval construction does not model the covariance between them. A verdict built on an interval that can read too narrow would overstate what the data support.
A flat table with one row per item, an item-id column and one verdict column per harness. CSV and TSV both work. With OpenAI output items as the source, the new harness's verdicts come in a flat table of their own, with an item-id column and one verdict column. Verdict tokens are case-insensitive: true, 1, yes, y, t, pass, passed for pass and false, 0, no, n, f, fail, failed for fail.
Use it when the baseline and the post-migration run scored the same items and you can match them row by row. The paired interval is narrower because it accounts for the correlation between items that appear in both runs. If the two runs cover different case sets, use the unpaired path with the summary counts.
No. The arithmetic value is shown as it is. A value above 100% or below 0% means the offset is large enough that adding it to the recorded baseline pushes the number past the boundary. It signals that something unexpected happened in the overlap, not a calculation fault.
Choose OpenAI output items as the source for the overlap run or the paired observed path. The page reads API output items as JSONL, a JSON array, one list page, list pages as JSONL lines, or a single item, and the dashboard's eval_items download, labeled provisional because OpenAI publishes no specification for it. The OpenAI items fill the old harness column on the overlap run and the baseline column on the paired observed path. Each item pairs with a row in your other file by a shared id field, pre-filled from "id", "case_id" or "item_id" and changeable. Repeated keys keep their first row, and unmatched items are counted; those with a key are listed.
An output item is unscored when its status is neither pass nor fail, or when its sample carried an error. The Unscored items control offers the three conventions the pass-rate confidence interval calculator uses. Scored only leaves unscored items out of the table. All attempted counts each one as a failure on the OpenAI side, and Bound pair, upper edge counts it as a pass, so those two put the OpenAI pass rate at the lowest and highest value the unscored items allow. With no unscored items, the three give the same table and the page says so. Unmatched items are always left out; the summary counts them and the page lists the first 20 that have a key.