Rows in A
20
INSTRUMENT | Reliability testing
Golden Set Version Tracker: Dataset Diff and Changelog
Diff two versions of your golden eval set in the browser, keep a dated changelog of what changed and why, and freeze the baseline metric values the next run is judged against.
An eval set that changes without a record makes score movement unreadable: system and set cannot be told apart. Load two versions, read what changed, write down why, freeze the baseline the next run is judged against. Reading a moved score grounds this page.
See also schema checks, split overlap, the reproducibility rule, and gamed benchmarks. Version labels may follow semver; changelog entries, Keep a Changelog. The changelog reuses the judge tracker's store; a run register cites the version.
1. Compare two versions
Showing your last valid result. Update the inputs above to recompute.
This is the bundled example: a 20-row version compared against a 21-row version, matched by row id. Load your own two files and it is replaced.
What changed
6
3 added, 2 removed, 1 relabeled, 0 changed. 20 rows in A, 21 in B, 17 unchanged.
Churn
30.0%
Added (3)
| Key | Input |
|---|---|
| 21 | Question 21 text |
| 22 | Question 22 text |
| 23 | Question 23 text |
Removed (2)
| Key | Input |
|---|---|
| 19 | Question 19 text |
| 20 | Question 20 text |
Changed (0)
| Key | Differing fields | Old value | New value |
|---|
Relabeled (1)
| Key | Differing fields | Old value | New value |
|---|---|---|---|
| 5 | expected | Answer 5 | Answer 5 relabeled |
Skipped from matching
| Skipped | Count |
|---|---|
| Blank keys | 0 |
| Duplicate ids | 0 |
| Repeated content keys | 0 |
Fingerprint A: 20:72cc43a2. Fingerprint B: 21:37fd1d94.
Not a security check: 32 bits catches a wrong or truncated file, nothing about tampering.
2. Record a changelog entry
Newest first.
| Date | Version | Kinds | Diff | Reason |
|---|---|---|---|---|
| No entries yet. | ||||
Stored on this device only.
3. Freeze a baseline
Manifest preview (Markdown)
No baseline frozen yet.
churn = (added + removed + changed + relabeled) / rowsA * 100How?
How this is calculated
A row's key comes from id when every row in both files carries a non-empty one
(tier 1). Otherwise, or if you pick it, the key is the row content on the fields you choose
(tier 2). Under tier 2 an edited key field reads as one removal plus one addition, because the
pairing itself is gone. The page always says which tier ran and why.
A row counts once: changed means input differs;
relabeled means input is identical but expected,
split, slice, or tags differ. Changed wins when both
apply, and lists which label fields also moved. The version fingerprint hashes every row's key
plus its compared fields, so a relabel under a stable id still moves the identity, a
same-version-or-not signal with no security guarantee.
Formula: churn = (added + removed + changed + relabeled) / rowsA * 100
Questions
Questions
What is the difference between changed and relabeled?
Changed means the input itself is different. Relabeled means the input is identical but the expected answer, split, slice, or tags moved. This is often the more common explanation for a score change, since it means the grading criteria shifted rather than the question.
Why did dropping the id column change my counts?
Without a usable id on every row, matching switches to row content. An edited value in the key field breaks that pairing, so the old row reads as removed and the new one as added, rather than as one changed row.
What happens to a duplicate id?
A repeating id is excluded from matching entirely and counted separately, so one stray duplicate in a 200,000-row file cannot turn every real edit into a removal plus an addition. Every other row still matches by id.
Does anything about either file leave my browser?
No. Both files are read and compared on this page. Only the changelog's counts and text are saved, never a row, a cell value, or a file name, and only on this device.