TOOL | Eval statistics
Write the eval readout for a launch decision
Collect the decision, every figure with its interval, method and source, the segments, residual risk and what was not tested, for whoever signs off. A judge-graded rate carries the judge's own errors; our research on bias-correcting a judge's pass rate shows the correction, and you enter the corrected rate as a reported row. The evaluation card records how an evaluation ran; this page records the decision. Export the readout or copy a summary into chat.
Load a paired comparison report from the eval A/B comparator or an eval results audit from the eval results auditor to fill figures and computed risk lines; typed rows stay.
Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.
JSON. Drag one here, or use the box below.
Check this value.
Reads an eval readout saved from this page as JSON, version 1.x. Restores the form, figures, segments, risk and not-tested lists, and the test report.
Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.
JSON. Drag one here, or use the box below.
Check this value.
A readout holds the decision, the figures with their intervals and methods, the segments, the residual risk, what was not tested and where each number came from. Load the returns-desk example to see what it looks like.
Where the headline's interval sits against the bar shows whether these cases put the headline clearly on one side of it at the stated level.
Storage is not available. The draft is not kept between visits. Export before navigating away.
How the readout is built
The page computes an interval from k of n using the stated method and confidence level, reads the headline against the bar, and prints every figure with its source.
Each counts row gets its interval from the method and confidence level. Wilson and Clopper-Pearson intervals are built through the same function the release gate designer uses. All intervals are two-sided.
Each reported row prints the value and the interval you bring. No interval is computed for it. The method name is the one you typed.
The bar applies to the headline row and to each segment; a segment whose whole interval sits on the wrong side is listed as a residual risk. The bar is read with inclusive edges (the bar itself counts as clearing: an at-least bar is cleared when the lower bound equals it, an at-most bar when the upper bound equals it), the same rule the release gate designer applies. When the interval contains the bar, the reading is across the bar: the true value could sit on either side, and more cases narrow the interval.
A segment with fewer than 30 cases is flagged. Thirty cases is this site's default floor for a rate claim; whether a smaller segment is enough depends on the rate you need to claim.
An import from a paired comparison report fills figures, segments and the cases that passed on the baseline and fail on the candidate. An import from an eval results audit fills figures and computed risk lines. Each imported row names its source from the file.
The summary follows a fixed clause order: the decision and the question; the headline figure with its interval and k of n; the bar reading; the first residual risk and the count of the rest; the count and the first line of what was not tested; and who decided and when. To fit 600 characters, the page drops who decided, then the first not-tested line, then the first risk's text (its count stays), then the bar reading. A question or label that still overflows is cut with an ellipsis, at a code-point boundary, and no number is split.
The test completion report is in use once any field is filled. A criterion not met, empty criteria, a problem or unchecked threat, and an unmeasured grader side each join the residual risk. Catch rate is caught over real failures; false alarm rate is false alarms over real passes. Each carries a two-sided 95% Wilson interval from the typed counts; the labeled set source prints with each rate in the document under "Where the numbers came from". Criteria print as counts. When the decision is Ship with open items, the page, the document and summary carry the count; the clause follows the risk clause and drops last.
Questions
What is the difference between this and the evaluation card?
The evaluation card documents one evaluation's method: what was tested, how many cases, which graders, what temperature, what parser. This page documents a decision: what the team decided, what the numbers were, where they came from, what was not tested and who signed it. A figure's source may name a card, and a readout may rest on several evaluations, each with its own card.
Why does each number carry an interval and a source?
An interval is the range the true value plausibly sits in, given the number of cases and the method named beside it. A source names where the number came from and when. Together they let a reader who was not in the room check the claim without running the eval again.
What does "across the bar" mean?
The interval around the headline contains the bar, so the true value could sit on either side. Running more cases on the same version narrows the interval. Until the interval falls wholly on one side of the bar, the data do not settle the question.
Why list what was not tested?
A readout that lists its numbers and says nothing about what was not covered reads stronger than it is. An empty not-tested section prints NOT RECORDED in the document, so that a reader sees the omission.
What is the difference between proposed and decided?
Proposed means the team's recommendation awaits sign-off. Decided means someone has signed it. The date field records when.
Can I paste the summary into a channel?
Copy the summary from the summary control. It fits in 600 characters and carries the decision, the headline with its interval, the bar reading, the first risk and the first not-tested line when they fit. The document holds every detail the summary leaves out.
Which files does it read?
The report control reads two kinds: a paired comparison report from the eval A/B comparator fills figures, segments and the cases that passed on the baseline and fail on the candidate; an eval results audit from the eval results auditor fills figures and computed risk lines. A saved readout opens through its own control and restores the whole form.
Does this page decide anything?
No. It records a decision. It reads the headline against the bar and says where the interval sits, and it lists the risk and the gaps. The decision is owned by the named decider, whoever typed it in.
How does the test completion report work?
The test completion report at the end of the form is for the decider. The preparer fills exit criteria, the grader's counts against a person's labels, deviations and threats. Either person may type the name in "Prepared by" or "Decided by". Use the evaluation card to record how one evaluation ran. Use the release gate designer to set the go or no-go rule. This page records typed statuses and computes no verdict. Adding repeats per case and the grader's error rates to a standard test completion report is our synthesis.