TOOL | Eval statistics
Claim-Evidence Matrix: What Backs Each Deployment Claim
Each row is a deployment claim: the artifacts, a per-dimension status across eleven evidence dimensions, and a named reason where limited. Residual deficits are listed, not scored. Built on the evidence quality framework and the methodology in reporting LLM-as-a-judge evaluations. No composite score or sufficiency verdict. To check one run, use what does this evaluation prove. A "B beats A" claim is a paired comparison on the same cases or a two-proportion comparison on separate sets.
Assess one claim
Evidence register
No claims registered yet. Use the form above to assess your first claim.
Adequate Limited Not recorded Not applicable
Worked example (static preview). With JavaScript, the register is interactive.
| Claim | Class | Rel | Repr | Meth | Stat | Cons | Repro | Trac | Ind | Rec | Comp | App | Deficits | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Intent routing pass rate on the September sample is 91% (95% interval 86 to 95, n = 180) | Capability on this set | + | ~ | + | ~ | + | ? | + | + | + | ~ | + | Repr: convenience sample, not production traffic; Stat: n = 180, wide interval on tail slice; Comp: refund path not tested |
Gaps and deficits
No gaps to show.
Accuracy and limitations statement
For a customer, a legal team or a sales team, declare the system, its accuracy with intervals, known failure modes, what was not tested, limits and monitoring. Linking a register claim carries its evidence appraisal into the statement. Export for a questionnaire or attach as a document.
This section declares accuracy for a deployed system and pulls limits from the register. The register's per-claim appraisal stays on the dossier export; the statement carries the outward summary.
These figures are the fictional returns-desk example. They do not change the register above.
A statement declares the system's accuracy with its interval, lists the known failure modes, what was not tested, limits of the evidence and monitoring in place. It draws on linked register claims. Load the returns-desk example to see what it looks like.
How the statement is assembled
Each declared figure is one of two kinds. A counts row gets a computed interval from k of n at the stated level using Wilson or Clopper-Pearson. A reported row prints the value and the interval you bring.
The eleven evidence dimensions on each linked claim map to three sections of the statement. Limited completeness, representativeness, applicability and recency go to what was not tested, because each describes a population, context or version the evidence did not cover. Limited relevance, methodological soundness, statistical adequacy, consistency, reproducibility, traceability and independence, and every residual deficit, go to limits of the evidence. Dimensions marked not recorded are listed under not yet assessed. Adequate and not-applicable dimensions print nothing.
An imported readout fills one declared figure, the headline. Of the lines entered by hand, the not-tested lines go to what was not tested and the risk lines go to limits of the evidence. The other figures, segments and computed risk lines stay in the readout. The section on where the figures come from says the figure was imported from an eval readout and gives the readout's date.
The readout page exports the file this section imports.
The statement makes no compliance claim and maps no clause to a regulation. Article 13(3)(b)(ii) of the EU AI Act requires that a high-risk system's instructions for use state its level of accuracy and the metrics behind it, and Article 15(3) requires that those levels and metrics be declared. Regulation (EU) 2026/1744 defers the application dates to December 2, 2027 for Annex III systems and August 2, 2028 for Annex I systems. The Sources section lists both regulations.
Reads an eval readout file (JSON, version 1.x). The headline figure is carried into the statement, along with any not-tested and risk lines entered by hand. The rest stays in the readout.
Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.
JSON. Drag one here, or use the box below.
Check this value.
Reads an accuracy limitations statement saved from this page as JSON. Restores the form, declared figures, failure modes and monitoring.
Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.
JSON. Drag one here, or use the box below.
Check this value.
The statement draft is stored on this device only.
Storage is not available. The statement draft is not kept between visits. Export before navigating away.
Eleven evidence dimensions
Each claim is assessed against eleven dimensions defined in the evidence quality framework. A dimension marked "limited" carries a written reason; "not recorded" flags a gap the reader has not yet inspected; "not applicable" means the dimension does not bear on this claim.
- Relevance
-
Does the evidence address this claim in this context of use?
Rate down when: Wrong population, wrong comparator, or wrong outcome for the claim.
- Representativeness
-
Was the sample drawn from the population the claim covers, including its tail?
Rate down when: Sample drawn from a narrower population than the claim addresses.
- Methodological soundness
-
Was the evaluator valid, were slices declared in advance, was grading auditable?
Rate down when: Evaluator not validated, post-hoc slicing, or unauditable scoring.
- Statistical adequacy
-
Is n sufficient for the precision claimed, per slice, with intervals reported?
Rate down when: Sample too small for the claimed precision, or intervals not reported.
- Consistency
-
Do independent runs, slices, and evaluators agree?
Rate down when: Runs, slices, or evaluators disagree beyond the reported interval.
- Reproducibility
-
Could someone else re-run it with versions, prompts, seeds, dates, and harness pinned?
Rate down when: Missing version pins, unpublished prompts, or undisclosed seeds.
- Traceability
-
Does each number trace to specific cases and a specific system version?
Rate down when: Aggregate without case-level traceability or unspecified system version.
- Independence
-
Who produced the evidence, and did they have an interest in the result?
Rate down when: Evidence produced by a party with a stake in the outcome.
- Recency
-
Does the evidence describe the currently deployed configuration?
Rate down when: Evidence predates the deployed version or a material configuration change.
- Completeness
-
What was not tested, stated rather than omitted?
Rate down when: Known gaps not stated, or selective reporting of favorable slices.
- Applicability
-
Does the deployment context match the evaluation context?
Rate down when: Evaluation context differs materially from the deployment context.
Admissible artifacts include a judge calibration log and a judge version card. The evidence pack feeds the release gate designer.
Export
Claim-evidence register
Deployment evidence dossier
Accuracy and limitations statement
Import
A previously exported register file. Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.
JSON. Drag one here, or use the box below.
Check this value.
Common questions
Why no composite score?
The eleven dimensions are not commensurable: good reproducibility does not offset an invalid evaluator. The evidence quality framework gives the full argument against a composite; the short version is that any weighting is a value judgment disguised as arithmetic, and a composite is a target that will be optimized.
What does "not recorded" mean versus "not applicable"?
"Not recorded" means you have not yet assessed that dimension for this claim. It is a visible gap, flagged in the deficits panel. "Not applicable" means the dimension genuinely does not bear on this claim and is not counted as a gap. The two are never interchangeable: a dimension you skipped because you ran out of time is "not recorded," not "not applicable."
What is a residual deficit?
A deficit that remains after you have assessed every dimension. It is a named doubt in the evidence base that you choose to carry rather than resolve, stated explicitly so a reader knows it exists. An assurance case without stated residual deficits is not making a stronger argument; it is making an incomplete one.
What is the difference between the dossier and this statement?
The deployment evidence dossier is the per-claim appraisal for an internal reviewer: eleven dimensions, the status grid, the symbols. The statement is the outward summary for a customer or legal team: declared accuracy, failure modes, what was not tested, limits and monitoring, with its limits drawn from the same register. It prints no status grid, no symbols and no score.
Why do limits come from linked register claims?
Each claim in the register carries an evidence appraisal with a status for every dimension and, where the team wrote one, a reason for each limited dimension. Linking a declared row to a claim pulls those reasons into the statement so that nothing the team recorded as a limit or a gap gets left out.
Does this statement certify compliance?
No. It declares accuracy with its interval, lists known failure modes and gaps, and names monitoring in place.