TOOL | Eval statistics
Is this eval number ready to quote?
Answer short questions about an eval result to get one of three verdicts. Quotable: the number has a case count, an interval, a checked grader, and no known gaps. Quotable with caveats: the number stands, with sentences to paste beside it. Not yet: a blocking gap must close first, and the page names it and the tool that closes it. For the claims a whole evaluation supports, use the page on what an evaluation proves.
Answer each item to get a verdict, or load the returns-desk example to see a finished check.
Your answers are processed in this browser tab, and you can check by going offline after the page loads. This page never uploads them.
How this is calculated
The verdict rule. Any unanswered item leaves the check with no verdict. Any blocking item answered no or not sure gives "not yet". Any caveat item answered no or not sure, or any figure warning, gives "quotable with caveats". Otherwise the verdict is "quotable".
Blocking items. Four items block: whether the number has a case count and interval, whether the grader was checked against human labels, whether an overlap check confirmed no leakage, and whether unscored attempts were counted. A not-sure answer on a blocking item produces "not yet" because a gate the reporter cannot vouch for is not passed.
The whole-cases check. The page counts the decimal places you typed and looks for a whole number of passes, from 0 up to the case count, whose rate lies within half a unit of the last decimal place you typed. If no whole number matches, the figure is not a whole number of cases out of the stated count, and the nearest candidates are shown.
The width check. The page computes the two-sided Wilson score interval at the stated case count and confidence level and compares its width with the stated interval's width; the check compares widths only. A ratio below 0.75 triggers a warning. Counting each repeat of a case as a separate case shrinks an interval's width to about one over the square root of the repeats per case, which is 0.71 at two repeats, so 0.75 catches two or more repeats per case. A Wilson or Clopper-Pearson interval computed on the cases themselves sits at or above 1.
Questions
What does "quotable with caveats" mean?
The number passes every blocking check, but one or more items call for a sentence beside it. The caveat sentences are ready to paste: they name what is missing or unconfirmed and what it means for the number's scope.
Why do some answers block the verdict?
Four items address facts that can make a number misleading enough to withdraw: no denominator and no interval leave the precision unknown, an unchecked grader may disagree with human labels, leaked test cases can inflate the rate, and dropped unscored attempts shrink the denominator. Reporting the number without resolving these gaps risks a claim the data does not support.
What happens when I answer "not sure"?
On a blocking item, "not sure" gives the same verdict as "no" because a gate the reporter cannot vouch for is not passed. On a caveat item, "not sure" produces a caveat sentence that says the fact is unconfirmed, so the reader knows to check it.
What do the figure checks catch?
The whole-cases check catches a figure that cannot come from any integer number of passes out of the stated case count, which points to a wrong case count or a figure rounded at a different precision. The width check catches an interval that is narrower than the case count supports, which points to repeats within a case counted as separate cases, inflating the denominator and shrinking the interval.
Why does the number of variants tried matter?
When two prompts were tried and the better one is quoted, a reader who sees only the quoted rate cannot tell how much of its margin came from picking the better of two. Disclosing the count lets the reader decide.
What about a comparison on different cases?
A comparison on different cases mixes case difficulty with version performance. If one version ran on harder cases, part of the gap comes from the cases. The checklist marks this as a caveat when the answer is no or not sure, and points to the paired comparison tools, and to the independent two-proportion page for when different cases are all you have.
Where are my answers kept?
When this browser allows storage, your answers are kept in it for this page and nowhere else. The answers survive a reload but do not sync between devices or browsers. Export the check as JSON for a durable record, and import it on another device or after clearing your browser data.
How does this differ from the page on what an evaluation proves?
The page on what an evaluation proves takes the evidence one evaluation produced and shows which claim classes it supports, with the tool that closes each gap. This checklist gates a single number for quoting: is the case count stated, is the grader checked, is leakage ruled out, are unscored attempts counted? Start here before a deck or readout; go there when you want the deeper appraisal.
After "not yet", what should I do next?
Each blocking reason names the gap and the tool that closes it. Fix the blocking items first. Once no blocking item is at no or not sure, the verdict moves to "quotable" or "quotable with caveats", and the caveat sentences are ready to paste beside the number.