INSTRUMENT | eval
Reproducibility Checklist and Manifest Validator
7 cited sources
Check whether your write-up records what someone else would need to rerun your eval: a checklist against the published card schema, and a validator for a pasted or uploaded manifest.
Before you publish a result, check what a second team needs to rerun it. Reproducibility starts with the record: 96.5 percent of evaluation results miss at least one minimal field, checked here against the fields the evaluation card generator publishes.
A complete manifest means the record is complete, never that the number reproduces: a temperature pinned at zero still drifted on two of seven items. Log runs in the run register; see our house methodology.
Showing your last valid result. Update the inputs above to recompute.
Items needing attention
Fill the checklist, or check a manifest, to see a verdict.
- Applicable
- 0
- Recorded
- 0
- Invalid
- 0
- Required missing
- 0
- Exempt with a reason
- 0
- Recommended gap
- 0
Ordered invalid, missing, exempt, recommended gap, recorded.
| Item | Status | Reason / why it matters | Source |
|---|
JSON carries the card and re-imports on this page. Markdown is the findings table, for reading in a plain editor. It is one way and is never re-read.
to fill by hand outside this page.
pass one: the published v1 schema (type, range, enum, date, group) · pass two: the reproducibility-minimal overlayHow?
How this is calculated
Two passes, never one. The published evaluation-card v1 schema judges every
type, range, enum, date, and grouped field (a judge or a harness needs both a name and a version, or
neither) and runs first. A value it rejects reads invalid, never missing and never
recorded, and the verdict cannot read complete while one stands. This profile adds no type rule of its
own and relaxes none the schema sets.
Only then, requiredness. Of the twenty items, nineteen are required; one, the dataset's SHA-256, is recommended. A value that cleared pass one but was never written down reads missing. An item the reader declares not applicable, with a reason, reads exempt; a blank reason still counts as exempt but prints as a warning, never an error.
Zero is a value. method.temperature: 0 reads recorded, never missing: the
most common real-world case a naive presence check gets wrong. A whitespace-only value, or a stand-in
like "N/A", "none", or "-", reads missing with a hint pointing at the not-applicable control instead.
Run spread leaves the applicable set only when a run count of exactly one says so. Every other reading of the run count (absent, invalid, two or more) keeps the item applicable, so nothing drops out silently just because a fact this checklist cannot verify was never confirmed.
The fingerprint. SHA-256 over the raw bytes of a picked file, computed with
crypto.subtle.digest in this browser: the same value shasum -a 256 prints at a
terminal. Compared against a manifest's data.sha256, a mismatch prints a warning naming it,
never an error: the two may be different files on purpose.
Formula: pass one: the published v1 schema (type, range, enum, date, group) · pass two: the reproducibility-minimal overlay
The reproducibility-minimal profile
Binds to the published evaluation-card v1 schema, pinned 2026-08-27, never to any TypeScript type. A requiredness change publishes a v2 profile beside this one, never over it.
The twenty reproducibility-minimal v1 items.
| Item | P-06 field | Required | Applies when | Why it matters |
|---|---|---|---|---|
| Model name | subject.name | required | always | A family name is not a system |
| Model version | subject.version | required | always | The resolved snapshot, and the date it resolved |
| Temperature | method.temperature | required | always | Absent from 93.9 percent of triples |
| Max tokens | method.maxTokens | required | always | Absent from 95.6 percent of triples |
| Seed | method.seed | required | the vendor exposes one | Pinned zero still drifted on 2 of 7 items |
| Harness | method.harness | required | a harness or scaffold ran | Named in the minimal sub-schema |
| Eval plan | method.evalPlan | required | agentic run | Missing from 100 percent of agentic triples |
| Eval limits | method.evalLimits | required | agentic run | Step, tool call, wall clock, or token budget. Missing from 100 percent |
| Dataset source | data.source | required | always | Which set was scored |
| Dataset version | data.version | required | always | Sets move, and M-T01 tracks that |
| Dataset SHA-256 | data.sha256 | recommended | a file exists | Proves which bytes were scored |
| Dataset size | data.size | required | always | Items scored, so a reader can size the uncertainty |
| Metric name | method.metric | required | always | The metric path is the third leg of the triple |
| Metric definition | task.construct | required | always | Two teams score the same output differently |
| Scoring parser | method.parser | required | always | How a raw output became a score |
| Judge | method.judge | required | a model graded | Grader config is part of the result |
| Run count | method.runs | required | always | One run is a sample of one |
| Run spread | a result.figures row whose intervalKind is 'run-spread' | required | method.runs recorded and above 1 | Report variance, never a lone point estimate. A single-run confidence interval is not a spread |
| Run date | provenance.runOn | required | always | A provider can move under a fixed version string, so a result has to be datable |
| Limitations | limits | required | always | The weakest section everywhere: 2.1 percent of card text |
Sources
- Evaluation Cards: measuring reproducibility, completeness, provenance and comparability across 101,843 results
- Temperature control and reproducibility in judge evaluations
- A 2024 study of completeness across 7,433 Hugging Face dataset cards
- The NeurIPS paper checklist's not-applicable-with-a-reason precedent
- REFORMS, a reporting standard built from one table rather than two documents
- The MLCommons Croissant metadata spec's per-file SHA-256 convention
- The Data Package standard's permissive-schema-plus-profile design