LatentEval

TOOL | Evaluation design

How much evaluation does this feature need?

Six questions about your feature determine how much evaluation it needs before launch: a spot review of real outputs, a fixed test set rerun on every change, measured rates against a bar, or pre-registered gates with validated graders. Each tier carries its minimum setup, drawing on what each kind of eval can certify. The answers that would change the verdict are named. Failure families, case classes and evidence expectations come from the context-of-use profiler.

Tiers are checked heaviest first, the first matching rule decides, and each tier's minimum setup carries every lighter tier's items.

Lost time or money: rework, a refund, a lost sale. Harm to a person: financial, legal, health, safety or privacy harm.

Check this value.

Who carries the consequence of a mistake: only the operator, internal colleagues, the end user, third parties, or a regulated population.

Check this value.

Whether a wrong action can be undone for free, made good with compensation, undone within a window, undone by hand, or not at all.

Check this value.

Who looks at outputs before they land: no one, a sample, every output, or a reviewer under time pressure.

Check this value.

How many outputs the feature produces a day. Choose "Not known yet" if unsure; the result names what this answer would decide.

Check this value.

How often the team changes the prompt, model or retrieval setup that feeds the feature.

Check this value.

If the team tracks a severe class, enter its expected rate as "1 in N" to see a per-day exposure count.

Check this value.

Answer these questions to see a verdict: Worst plausible harm from one bad output, Who carries the consequence, Whether the action can be reversed, Whether a person reviews each output. No answer is defaulted for you.

Export the verdict
How this is calculated

The page applies ordered rules to the six answers, heaviest first, and the first matching rule sets the tier. The rule table is a LatentEval synthesis of evaluation practices. Only the first matching rule counts, and no answers are averaged or scored.

Each tier carries the lighter tiers' items. Their numbers come from standard formulas: the chance of seeing a failure at least once (99 outputs), the inverted Wilson interval (245 cases), Connor's paired sample size (870 paired cases) and the zero-failure count for a one-sided bound (59 cases).

To find the answers that would change the verdict, the page swaps in each other option for one question at a time, holding the rest fixed, and reports every swap that moves the tier. The headline shows at most one change per direction (the first in form order); a fold shows all of them.

The exposure line multiplies the volume band's lower edge by a failure rate (1 in 1,000, 1 in 10,000, or your own severe-failure rate) to give a per-day count of expected failures (for under 100 a day, the upper edge, printed as fewer than). This is arithmetic on the entered volume.

Questions

Why does this page give a tier and not a score?

A score would need weights across the six questions, and those weights would be arbitrary: possible harm to a person matters more than volume, and an aggregate number hides that ordering. The page returns one of four tiers, each carrying a cumulative list of practices with numbers that size them. Spot review is the floor; each higher tier adds its own items. The tier names what changes in the setup, and the answers that would change it show how sensitive the verdict is.

Why is the first question about harm?

Seven of the nine rules that escalate the tier read the harm answer, more than any other question. When the worst plausible outcome is harm to a person, the verdict is at least measured rates against a bar, because a single bad output can have consequences that volume and reversibility cannot offset. Asking harm first means you see the most influential question first.

What happens when I answer "not known yet"?

For a required question, the page shows no verdict until all four are answered. For an optional question (how often it runs and how often the setup changes), the page gives the best verdict it can and names what each unanswered question would have decided: volume decides whether a rare failure is frequent enough to matter at scale, and change cadence decides whether a fixed set needs to track each change.

How does this page differ from the context-of-use profiler?

The profiler maps eleven dimensions of your feature into six categories of written items (failure families, case classes and evidence expectations among them), all qualitative. This page asks six questions, four shared with the profiler, and returns a tier with a sized setup list. Two of its six questions are new to this page: worst plausible harm and how often the setup changes.

How does this relate to the agent-or-workflow decision page?

That page decides whether the feature should be an agent, a fixed workflow, or no model at all, and for an agent feature it comes first. Its four shape questions (whether the steps can be written down in advance, whether a person could write the rule, whether the systems it acts on expose callable interfaces, and how many steps the model chooses) do not appear here. The depth question on this page applies to either shape.

What do I give up by going one tier lighter?

Each tier drops the items that only the heavier tier carries. Going from gated to measured rates drops the frozen gate rule, the judge validated against human labels, the sized zero-failure claim, the named signer and monitoring after launch. Going down to a fixed set drops the bar set in advance, the intervals, the paired check, the separate severe count and the grader check. Going down to a spot review drops the rerun on every change; outputs are still read by hand before launch and after each change.