Eval Run Register: Preregistration and Run Cards for LLM Evals

Verdict. Read the subject variant's reading whose metric matches the primary metric, and compare it to the threshold with the chosen rule (at least, at most, greater than, less than). No rule chosen yields "inconclusive" and the reader sets a verdict by hand with a reason. Numbers compare raw, never rounded first. One threshold on one metric is a reading, not a shipping decision: where several rules have to agree before anything goes out, write them down in the rule set that turns figures into promote, hold or roll back and cite this entry as the evidence behind one of them.

Plan fingerprint. The plan is turned into one deterministic string (sorted fields, collapsed whitespace, variants sorted by their id so reordering them on screen is not a change) and hashed with SHA-256. Freezing stores that hash plus a full copy of the plan. The dataset version in that plan is a string this page takes on trust, so it is worth citing one you can look up later: what actually changed between two versions of an eval set is a separate record, and a plan that names a version nobody kept is a plan nobody can rerun.

Plan drift. Every later save recomputes the fingerprint over the live plan. A mismatch names exactly which fields changed since the freeze, by comparing the live plan to the frozen copy field by field. Once flagged, an entry stays flagged, even if the edit is reverted.

Baseline difference. Subject value minus baseline value on the primary metric, shown exactly as entered. It is arithmetic, not a significance test, and the page never claims otherwise.

Missing-field rate. Over completed entries only, the share missing at least one of seven fields a re-run needs (model id, model version, prompt version, temperature, max tokens, dataset version, and the sample size behind the primary reading), to one decimal place.