LatentEval

TOOL | Eval statistics

Evaluator and Metric Navigator: Which Grader, Which Number

Answer four questions and get the evaluator type that is admissible for your oracle, the ones that are not and why, the metric family, and which tool runs the number. For the comparison-design question specifically, use the paired vs. independent design chooser. The oracle-evaluator admissibility rules are documented in the evaluation case schema. Built on the significance walkthrough.

A set built from any pack carries several oracle types across its families, so the navigator runs once per family and acceptance criteria are written per family.

The constraint matrix is our own synthesis of the admissibility rules implicit in the evaluation literature. It is ours, not a standard.

Your oracle type decides which evaluators are admissible, drawn here for a rubric; this admissibility matrix is our own synthesis.
What form of correct answer do you have?

Check this value.

Who is available to grade?

Check this value.

How many systems are you measuring?

Check this value.

Refine which grader takes which cases

Both questions are optional. "How much can a code check decide?" sets which cases code takes and appears for a reference answer, an acceptable set or a rubric. "How is the verdict used?" adds notes and, for live monitoring, limits people to a sample.

How is the verdict used?

Adds notes on what to watch, and for live monitoring limits a person to a sample; the grader stays as the other answers set it.

Check this value.

Oracle type x evaluator type: admissibility
Oracle type Exact or programmatic checkRubric-scored humanLLM judgePairwise preferenceTrajectory or state checkDomain expert
Reference answer Admissible Admissible With conditions Inadmissible Inadmissible Admissible
Acceptable set Admissible Admissible With conditions Inadmissible Inadmissible Admissible
Rubric Inadmissible Admissible With conditions Inadmissible Inadmissible Admissible
Expected tool calls Admissible Inadmissible Inadmissible Inadmissible Admissible Admissible
Expected final state Admissible Inadmissible Inadmissible Inadmissible Admissible Admissible
Expected behavior only Inadmissible Inadmissible Inadmissible Inadmissible Inadmissible Admissible

Comparison design x metric family: routing

RuleWhen it firesMetricRoute
M1 two systems scored the same cases Paired difference paired-vs-independent-design-chooser
M2 two systems scored different cases Pass rate with an interval independent-two-proportion-calculator
M3 one system measured alone, and every attempt must succeed pass^k reliability-at-k-estimator
M4 one system measured alone, and at least one attempt needs to succeed pass@k reliability-at-k-estimator
M5 one system measured alone Pass rate with an interval pass-rate-ci-calculator

Evaluator

LLM judge

Metric

Pass rate with an interval

Quality is multi-dimensional and written as criteria. With no human graders, a model judge is the admissible automated alternative, with its own agreement evidence.

The share of cases that passed, reported with a confidence interval on the share.

Which grader takes which cases

Each row is one grader, in the order they run. The Grades column says which cases it takes: every case, what code can decide, the rest, a sample that checks another grader, or a sample of live traffic.

LayerGradesCost per caseSame verdict twiceValidationWhat goes wrong
LLM judge Every case One model call per case, more with repeats. Can change between runs. Pin the model, prompt and settings and repeat it to measure how far its verdicts move. Per-class agreement with human labels on your own cases. Favors an answer's position or length, or its own model's outputs, and drifts with model changes.
Rubric-scored human A sample A person's time per case. People vary. Measure agreement between two on a sample. Chance-corrected agreement between two raters on a sample. Drifts and tires over long sessions.

The judge is a candidate until a labeled sample exists. Grade a sample of your own cases with a person, then measure per-class agreement.

What would change the grader

Each row is one answer, or a group of answers to the same question, that would change the recommended grader. These answers give a judge, not yet validated.

QuestionChange this answer toThe grader becomes
What form of correct answer do you have? "A reference answer is written down" or "A set of acceptable outputs is enumerated" or "The correct tool calls are known" or "The expected final state is checkable" code for every case
What form of correct answer do you have? "Only expected behavior is described" an expert (unavailable)
Who is available to grade? "Human graders on a sample" a judge
Who is available to grade? "Human graders for every case" a person
Who is available to grade? "A domain expert" an expert
How much can a code check decide? "Some of it" code first, then a judge, not yet validated

The two sections above name which grader takes which cases and what would change it. Sizing the labeled sample, computing a score and checking the judge class by class are separate steps.

Evaluators that are not admissible for this oracle type

Inadmissible evaluators

EvaluatorWhat it is
Exact or programmatic check Code decides. String match after a stated normalization, a parser, a schema check, a set membership test.
Pairwise preference A reader or a judge picks between two outputs. Inadmissible as an oracle-correctness evaluator because it answers which of two outputs is preferred, not which is correct. Where the claim itself is a preference or quality ranking, pairwise is the right instrument.
Trajectory or state check The path or the world after: which tools were called, what state changed, which constraint was violated.

Quality here is multi-dimensional and written as criteria. A string comparison cannot read a criterion. A trajectory check inspects the path, not the quality of the output.

Evaluators admissible with conditions

Conditionally admissible evaluators

EvaluatorConditions
LLM judge Admissible only with agreement evidence against human labels on your own items. Route to judge-validation-report-builder.
Validity evidence the chosen evaluator needs
  • Agreement against human labels: Measure the judge against human labels on your own items, not a published benchmark.
  • Position sensitivity: Swap the order of candidates and check whether the verdict changes.
  • Length bias: Check whether the judge favors longer outputs independently of quality.
  • Self-preference: Check whether the model prefers its own outputs over equally good alternatives.

Run each check on a sample of your own cases before trusting the grades.

Every rule we checked, and where each one stopped
StageRuleWhat happenedStopped at
1 E1 Stopped the oracle is a reference answer
1 E2 Stopped the oracle is an acceptable set
1 E3 Fired
1 E4 Stopped human graders are available on a sample
1 E5 Stopped human graders are available for every case
1 E6 Stopped a domain expert is available
1 E7 Stopped the oracle is expected tool calls
1 E8 Stopped the oracle is the expected final state
1 E9 Stopped the oracle is expected behavior only
2 M1 Stopped two systems scored the same cases
2 M2 Stopped two systems scored different cases
2 M3 Stopped every attempt must succeed
2 M4 Stopped at least one attempt needs to succeed
2 M5 Fired
How this is calculated

Two passes, not one. The first pass reads the oracle type and who is available to grade, and returns the evaluator. The second reads how many systems and what happens on failure, and returns the metric family plus a route to the live tool that computes it. Both are first-match rule tables: every rule is checked, the first that matches fires, and the rest are recorded as beaten or stopped.

Pass one: the evaluator

RuleWhen it firesEvaluator
E1 the oracle is a reference answer exact-or-programmatic
E2 the oracle is an acceptable set exact-or-programmatic
E3 the oracle is a rubric, and no human graders are available llm-judge
E4 the oracle is a rubric, and human graders are available on a sample llm-judge
E5 the oracle is a rubric, and human graders are available for every case rubric-human
E6 the oracle is a rubric, and a domain expert is available human-expert
E7 the oracle is expected tool calls trajectory-or-state-check
E8 the oracle is the expected final state trajectory-or-state-check
E9 the oracle is expected behavior only human-expert

Pass two: the metric

RuleWhen it firesMetricRoute
M1 two systems scored the same cases Paired difference paired-vs-independent-design-chooser
M2 two systems scored different cases Pass rate with an interval independent-two-proportion-calculator
M3 one system measured alone, and every attempt must succeed pass^k reliability-at-k-estimator
M4 one system measured alone, and at least one attempt needs to succeed pass@k reliability-at-k-estimator
M5 one system measured alone Pass rate with an interval pass-rate-ci-calculator

Admissibility. The constraint matrix maps each oracle type to the evaluators it admits and the ones it rules out. Pairwise preference is inadmissible as an oracle-correctness evaluator for every oracle type here, because it measures which of two outputs a reader prefers, not whether either is correct against the oracle. Where selecting between candidates is the actual question, the pairwise win rate calculator's protocol chooser settles pointwise versus pairwise.

Validity evidence. Some evaluators need their own proof that the grading is sound. A model judge needs agreement against human labels on your own items, position-sensitivity, length-bias, and self-preference checks. A rubric-scored human evaluator needs inter-rater agreement corrected for chance. A domain expert needs at least two experts agreeing on a sample. An exact-or-programmatic evaluator needs none: the code is the definition.

Apart from the two passes, the grader stack (the "Which grader takes which cases" table) reads the oracle type, who can grade, how much of the criterion a code check can decide, and how the verdict is used. Five rules apply in order. (1) Expected tool calls or expected final state: code decides every case. (2) Expected behavior only: a domain expert grades every case. (3) A reference answer or acceptable set: code when it can decide all; a judge or person when it can decide none; code first, then the judge or person for the rest, when it can decide some. (4) A rubric: the same split, except that code for all is a conflict. (5) How the verdict is used adds notes, and for live monitoring it limits a person or expert to a sample of live traffic. Who can grade sets the grader for the remainder: with no human graders, a model judge that stays a candidate until a person labels a sample; with graders on a sample, a model judge checked by a person on a sample; with graders for every case, a person, checked by a second person; with a domain expert, the expert, checked by a second expert. The "What would change the grader" table counts changes to the code layer, the main grader or admissibility only.

Which metric fits which task type
Task typeMetricGraderPitfall
Classification or routing Per-class precision and recall, macro-F1 Code Accuracy hides minority-class failure
Short-answer QA Normalized exact match, token F1 Code A paraphrase fails exact match
Summarization A validated judge's rubric pass rate; ROUGE-L as a smoke test only Judge (validated) ROUGE-L penalizes rewording that a person accepts
Translation chrF, or BLEU with its settings stated Code BLEU is sensitive to tokenization and brevity
Retrieval Recall at k, nDCG at k, MRR at the k the prompt uses Code A k larger than the prompt uses counts documents the model never sees
Agent task End-state success, pass^k (succeeds every time in k tries), cost and steps per success Code plus a judge or person for open-ended criteria The agent may reach the goal by an unsafe path
Structured output Schema-valid rate, field accuracy Code Output that passes the schema can still hold a wrong value
Safety Harmful-compliance rate, over-refusal rate on a benign set Code for format; a judge or person for substance Over-refusal hides behind a low harmful-compliance rate
What overlap scores reward and miss
MetricWhat it countsRewardsMisses
Exact match (after a stated normalization) A score of 1 when the normalized strings are identical, 0 otherwise Identical answers after normalization Any correct answer written differently
Token F1 (bag of tokens, SQuAD normalization) Harmonic mean of token-level precision and recall Partial credit for overlapping content Ignores word order; a wrong answer that shares many words with the reference scores well
BLEU (clipped n-gram precision to 4, brevity penalty) Geometric mean of clipped n-gram precisions times a brevity penalty Matching word sequences in the reference's order Sensitive to tokenization; one reference gives noisy scores; the brevity penalty can dominate short outputs
ROUGE-L (longest common subsequence F1) F1 of the longest common subsequence The same content in roughly the same order Penalizes valid rewording; misses factual errors
chrF (character n-gram F with beta = 2) Character-level n-gram F-score, recall weighted twice as heavily Matching character sequences, so inflected forms earn partial credit Still a surface measure; misses semantic equivalence
Questions

Questions

Why is pairwise preference inadmissible?

Pairwise preference measures which of two outputs a reader prefers, not whether either is correct against the oracle. It answers a different question. Where selecting between candidates is the actual question, the pairwise win rate calculator's protocol chooser is the right tool.

What if no evaluator is admissible?

This happens when the oracle is expected behavior only and no human grader is available. The path forward is to turn the behavior description into a rubric, an acceptable set, or a set of expected calls, which opens the door to other evaluators.

Why does the metric depend on the retry model?

pass^k, pass@k, and plain pass rate answer different reliability questions. pass^k (every attempt must succeed) falls with k; pass@k (at least one must succeed) rises with k. The retry model picks the one that matches what the product actually experiences.

What is the trace table for?

The trace shows every rule the engine checked, which one fired, and where each non-firing rule stopped. It is the audit trail: if the recommendation looks wrong, the trace tells you which clause made it so.

Does this tool run the metric?

No. It names the evaluator type and the metric family and links to the tool that runs the number. Each route in the recommendation block is a link to the next step.

Why does the navigator recommend code first and a judge for the rest?

Code checks are repeatable, fast and cheap to run. Their usual failure is a correct answer written in a form the check does not expect. Sending what code can handle to a judge wastes money and adds noise. Code takes the cases it can decide, and the judge or person takes only what remains.

What does "What would change the grader" show?

Each row names an answer, or a group of answers to one question, and the grader that would result. A row appears when the code layer, the main grader or whether the grader can be used (a judge validated, an expert available) would change. How the verdict is used adds notes and, for live monitoring, limits a person to a sample; the grader stays as the other answers set it, so that question never forms a row.

Which overlap metric should I use?

For short-answer QA, use exact match or token F1. For translation, use chrF or BLEU with its tokenization and reference count stated. For summaries, use ROUGE-L only as a smoke test; a validated judge's rubric pass rate is the primary metric. The fold "What overlap scores reward and miss" lists what each metric rewards and what it misses.

Sources

  1. Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaNeurIPS 2023 Datasets and Benchmarks (Zheng et al.) Retrieved