LatentEval

TOOL | Eval statistics

Claim-Evidence Matrix: What Backs Each Deployment Claim

Each row is a deployment claim: the artifacts, a per-dimension status across eleven evidence dimensions, and a named reason where limited. Residual deficits are listed, not scored. Built on the evidence quality framework and the methodology in reporting LLM-as-a-judge evaluations. No composite score or sufficiency verdict. To check one run, use what does this evaluation prove. A "B beats A" claim is a paired comparison on the same cases or a two-proportion comparison on separate sets.

One claim is rated across eleven dimensions; only the limited and not recorded ones produce named deficits, with nothing summed into a score.

Assess one claim

The claim

Check this value.

Narrows which evidence elements matter most.

Check this value.

Check this value.

Check this value.

Evidence dimensions

Does the evidence address this claim in this context of use?

Check this value.

Was the sample drawn from the population the claim covers, including its tail?

Check this value.

Was the evaluator valid, were slices declared in advance, was grading auditable?

Check this value.

Is n sufficient for the precision claimed, per slice, with intervals reported?

Check this value.

Do independent runs, slices, and evaluators agree?

Check this value.

Could someone else re-run it with versions, prompts, seeds, dates, and harness pinned?

Check this value.

Does each number trace to specific cases and a specific system version?

Check this value.

Who produced the evidence, and did they have an interest in the result?

Check this value.

Does the evidence describe the currently deployed configuration?

Check this value.

What was not tested, stated rather than omitted?

Check this value.

Does the deployment context match the evaluation context?

Check this value.

Named doubts in the evidence base you choose to carry, one per line. They appear in the register and, when you link the claim to a declared row, in the statement.

Check this value.

Evidence register

Adequate Limited Not recorded Not applicable

Worked example (static preview). With JavaScript, the register is interactive.

Claim Class Rel Repr Meth Stat Cons Repro Trac Ind Rec Comp App Deficits
Intent routing pass rate on the September sample is 91% (95% interval 86 to 95, n = 180) Capability on this set + ~ + ~ + ? + + + ~ + Repr: convenience sample, not production traffic; Stat: n = 180, wide interval on tail slice; Comp: refund path not tested

Gaps and deficits

No gaps to show.

Accuracy and limitations statement

For a customer, a legal team or a sales team, declare the system, its accuracy with intervals, known failure modes, what was not tested, limits and monitoring. Linking a register claim carries its evidence appraisal into the statement. Export for a questionnaire or attach as a document.

This section declares accuracy for a deployed system and pulls limits from the register. The register's per-claim appraisal stays on the dossier export; the statement carries the outward summary.

The deployed system this statement covers. Required for export.

Check this value.

The use, users, languages, inputs and conditions the system is designed for. Required for export.

Check this value.

Declared accuracy

Each row states one measured figure for the system. Counts rows get a computed interval. Reported rows carry the interval you bring. Link a row to a register claim to pull its evidence appraisal into the statement.

LabelKindCount (k)Total (n)Interval methodValueLowerUpperUnitLevelName of your methodVersionTest setMeasured onConditionsSourceLinked claim Actions

Conditions The conditions under which this figure was measured, such as the test set, the grader, or the language.

Linked claim A claim from the register above. Linking pulls limited dimensions and deficits into the statement.

Known failure modes

Each row names a known failure mode, its observed frequency if known, and how it is handled.

DescriptionSeen (k)Of (n)Handling Actions
What was not tested

Populations, conditions or versions the evidence did not cover. Linking a claim adds lines from its limited completeness, representativeness, applicability and recency dimensions.

Monitoring in place

Each row names a signal being monitored, how often, and what triggers a response.

SignalCadenceTrigger Actions

A role or email for questions about this statement.

Check this value.

The date of this statement. Blank uses the export date.

Check this value.

A statement declares the system's accuracy with its interval, lists the known failure modes, what was not tested, limits of the evidence and monitoring in place. It draws on linked register claims. Load the returns-desk example to see what it looks like.

How the statement is assembled

Each declared figure is one of two kinds. A counts row gets a computed interval from k of n at the stated level using Wilson or Clopper-Pearson. A reported row prints the value and the interval you bring.

The eleven evidence dimensions on each linked claim map to three sections of the statement. Limited completeness, representativeness, applicability and recency go to what was not tested, because each describes a population, context or version the evidence did not cover. Limited relevance, methodological soundness, statistical adequacy, consistency, reproducibility, traceability and independence, and every residual deficit, go to limits of the evidence. Dimensions marked not recorded are listed under not yet assessed. Adequate and not-applicable dimensions print nothing.

An imported readout fills one declared figure, the headline. Of the lines entered by hand, the not-tested lines go to what was not tested and the risk lines go to limits of the evidence. The other figures, segments and computed risk lines stay in the readout. The section on where the figures come from says the figure was imported from an eval readout and gives the readout's date.

The readout page exports the file this section imports.

The statement makes no compliance claim and maps no clause to a regulation. Article 13(3)(b)(ii) of the EU AI Act requires that a high-risk system's instructions for use state its level of accuracy and the metrics behind it, and Article 15(3) requires that those levels and metrics be declared. Regulation (EU) 2026/1744 defers the application dates to December 2, 2027 for Annex III systems and August 2, 2028 for Annex I systems. The Sources section lists both regulations.

Reads an eval readout file (JSON, version 1.x). The headline figure is carried into the statement, along with any not-tested and risk lines entered by hand. The rest stays in the readout.

Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.

JSON. Drag one here, or use the box below.

Check this value.

Reads an accuracy limitations statement saved from this page as JSON. Restores the form, declared figures, failure modes and monitoring.

Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.

JSON. Drag one here, or use the box below.

Check this value.

Eleven evidence dimensions

Each claim is assessed against eleven dimensions defined in the evidence quality framework. A dimension marked "limited" carries a written reason; "not recorded" flags a gap the reader has not yet inspected; "not applicable" means the dimension does not bear on this claim.

Relevance

Does the evidence address this claim in this context of use?

Rate down when: Wrong population, wrong comparator, or wrong outcome for the claim.

Representativeness

Was the sample drawn from the population the claim covers, including its tail?

Rate down when: Sample drawn from a narrower population than the claim addresses.

Methodological soundness

Was the evaluator valid, were slices declared in advance, was grading auditable?

Rate down when: Evaluator not validated, post-hoc slicing, or unauditable scoring.

Statistical adequacy

Is n sufficient for the precision claimed, per slice, with intervals reported?

Rate down when: Sample too small for the claimed precision, or intervals not reported.

Consistency

Do independent runs, slices, and evaluators agree?

Rate down when: Runs, slices, or evaluators disagree beyond the reported interval.

Reproducibility

Could someone else re-run it with versions, prompts, seeds, dates, and harness pinned?

Rate down when: Missing version pins, unpublished prompts, or undisclosed seeds.

Traceability

Does each number trace to specific cases and a specific system version?

Rate down when: Aggregate without case-level traceability or unspecified system version.

Independence

Who produced the evidence, and did they have an interest in the result?

Rate down when: Evidence produced by a party with a stake in the outcome.

Recency

Does the evidence describe the currently deployed configuration?

Rate down when: Evidence predates the deployed version or a material configuration change.

Completeness

What was not tested, stated rather than omitted?

Rate down when: Known gaps not stated, or selective reporting of favorable slices.

Applicability

Does the deployment context match the evaluation context?

Rate down when: Evaluation context differs materially from the deployment context.

Admissible artifacts include a judge calibration log and a judge version card. The evidence pack feeds the release gate designer.

Export

Claim-evidence register

Export register

Deployment evidence dossier

Export dossier

Accuracy and limitations statement

Export statement

Import

A previously exported register file. Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.

JSON. Drag one here, or use the box below.

Check this value.

Common questions

Why no composite score?

The eleven dimensions are not commensurable: good reproducibility does not offset an invalid evaluator. The evidence quality framework gives the full argument against a composite; the short version is that any weighting is a value judgment disguised as arithmetic, and a composite is a target that will be optimized.

What does "not recorded" mean versus "not applicable"?

"Not recorded" means you have not yet assessed that dimension for this claim. It is a visible gap, flagged in the deficits panel. "Not applicable" means the dimension genuinely does not bear on this claim and is not counted as a gap. The two are never interchangeable: a dimension you skipped because you ran out of time is "not recorded," not "not applicable."

What is a residual deficit?

A deficit that remains after you have assessed every dimension. It is a named doubt in the evidence base that you choose to carry rather than resolve, stated explicitly so a reader knows it exists. An assurance case without stated residual deficits is not making a stronger argument; it is making an incomplete one.

What is the difference between the dossier and this statement?

The deployment evidence dossier is the per-claim appraisal for an internal reviewer: eleven dimensions, the status grid, the symbols. The statement is the outward summary for a customer or legal team: declared accuracy, failure modes, what was not tested, limits and monitoring, with its limits drawn from the same register. It prints no status grid, no symbols and no score.

Why do limits come from linked register claims?

Each claim in the register carries an evidence appraisal with a status for every dimension and, where the team wrote one, a reason for each limited dimension. Linking a declared row to a claim pulls those reasons into the statement so that nothing the team recorded as a limit or a gap gets left out.

Does this statement certify compliance?

No. It declares accuracy with its interval, lists known failure modes and gaps, and names monitoring in place.

Sources

  1. Regulation (EU) 2024/1689 (AI Act), Art. 13(3)(b)(ii) and Art. 15(3); application dates as amended by Regulation (EU) 2026/1744European Parliament and Council Retrieved