Evaluator
LLM judge
TOOL | Eval statistics
Answer four questions and get the evaluator type that is admissible for your oracle, the ones that are not and why, the metric family, and which tool runs the number. For the comparison-design question specifically, use the paired vs. independent design chooser. The oracle-evaluator admissibility rules are documented in the evaluation case schema. Built on the significance walkthrough.
A set built from any pack carries several oracle types across its families, so the navigator runs once per family and acceptance criteria are written per family.
The constraint matrix is our own synthesis of the admissibility rules implicit in the evaluation literature. It is ours, not a standard.
| Oracle type | Exact or programmatic check | Rubric-scored human | LLM judge | Pairwise preference | Trajectory or state check | Domain expert |
|---|---|---|---|---|---|---|
| Reference answer | Admissible | Admissible | With conditions | Inadmissible | Inadmissible | Admissible |
| Acceptable set | Admissible | Admissible | With conditions | Inadmissible | Inadmissible | Admissible |
| Rubric | Inadmissible | Admissible | With conditions | Inadmissible | Inadmissible | Admissible |
| Expected tool calls | Admissible | Inadmissible | Inadmissible | Inadmissible | Admissible | Admissible |
| Expected final state | Admissible | Inadmissible | Inadmissible | Inadmissible | Admissible | Admissible |
| Expected behavior only | Inadmissible | Inadmissible | Inadmissible | Inadmissible | Inadmissible | Admissible |
Comparison design x metric family: routing
| Rule | When it fires | Metric | Route |
|---|---|---|---|
| M1 | two systems scored the same cases | Paired difference | paired-vs-independent-design-chooser |
| M2 | two systems scored different cases | Pass rate with an interval | independent-two-proportion-calculator |
| M3 | one system measured alone, and every attempt must succeed | pass^k | reliability-at-k-estimator |
| M4 | one system measured alone, and at least one attempt needs to succeed | pass@k | reliability-at-k-estimator |
| M5 | one system measured alone | Pass rate with an interval | pass-rate-ci-calculator |
Showing your last valid result. Update the inputs above to recompute.
The answers are set for the returns-desk example's reply criterion, whose correct answer is a written rubric.
These settings came from the link you opened.
Evaluator
LLM judge
Metric
Pass rate with an interval
Quality is multi-dimensional and written as criteria. With no human graders, a model judge is the admissible automated alternative, with its own agreement evidence.
The share of cases that passed, reported with a confidence interval on the share.
Which grader takes which cases
Each row is one grader, in the order they run. The Grades column says which cases it takes: every case, what code can decide, the rest, a sample that checks another grader, or a sample of live traffic.
| Layer | Grades | Cost per case | Same verdict twice | Validation | What goes wrong |
|---|---|---|---|---|---|
| LLM judge | Every case | One model call per case, more with repeats. | Can change between runs. Pin the model, prompt and settings and repeat it to measure how far its verdicts move. | Per-class agreement with human labels on your own cases. | Favors an answer's position or length, or its own model's outputs, and drifts with model changes. |
| Rubric-scored human | A sample | A person's time per case. | People vary. Measure agreement between two on a sample. | Chance-corrected agreement between two raters on a sample. | Drifts and tires over long sessions. |
The judge is a candidate until a labeled sample exists. Grade a sample of your own cases with a person, then measure per-class agreement.
What would change the grader
Each row is one answer, or a group of answers to the same question, that would change the recommended grader. These answers give a judge, not yet validated.
| Question | Change this answer to | The grader becomes |
|---|---|---|
| What form of correct answer do you have? | "A reference answer is written down" or "A set of acceptable outputs is enumerated" or "The correct tool calls are known" or "The expected final state is checkable" | code for every case |
| What form of correct answer do you have? | "Only expected behavior is described" | an expert (unavailable) |
| Who is available to grade? | "Human graders on a sample" | a judge |
| Who is available to grade? | "Human graders for every case" | a person |
| Who is available to grade? | "A domain expert" | an expert |
| How much can a code check decide? | "Some of it" | code first, then a judge, not yet validated |
The two sections above name which grader takes which cases and what would change it. Sizing the labeled sample, computing a score and checking the judge class by class are separate steps.
Inadmissible evaluators
| Evaluator | What it is |
|---|---|
| Exact or programmatic check | Code decides. String match after a stated normalization, a parser, a schema check, a set membership test. |
| Pairwise preference | A reader or a judge picks between two outputs. Inadmissible as an oracle-correctness evaluator because it answers which of two outputs is preferred, not which is correct. Where the claim itself is a preference or quality ranking, pairwise is the right instrument. |
| Trajectory or state check | The path or the world after: which tools were called, what state changed, which constraint was violated. |
Quality here is multi-dimensional and written as criteria. A string comparison cannot read a criterion. A trajectory check inspects the path, not the quality of the output.
Conditionally admissible evaluators
| Evaluator | Conditions |
|---|---|
| LLM judge | Admissible only with agreement evidence against human labels on your own items. Route to judge-validation-report-builder. |
Run each check on a sample of your own cases before trusting the grades.
| Stage | Rule | What happened | Stopped at |
|---|---|---|---|
| 1 | E1 | Stopped | the oracle is a reference answer |
| 1 | E2 | Stopped | the oracle is an acceptable set |
| 1 | E3 | Fired | |
| 1 | E4 | Stopped | human graders are available on a sample |
| 1 | E5 | Stopped | human graders are available for every case |
| 1 | E6 | Stopped | a domain expert is available |
| 1 | E7 | Stopped | the oracle is expected tool calls |
| 1 | E8 | Stopped | the oracle is the expected final state |
| 1 | E9 | Stopped | the oracle is expected behavior only |
| 2 | M1 | Stopped | two systems scored the same cases |
| 2 | M2 | Stopped | two systems scored different cases |
| 2 | M3 | Stopped | every attempt must succeed |
| 2 | M4 | Stopped | at least one attempt needs to succeed |
| 2 | M5 | Fired |
Two passes, not one. The first pass reads the oracle type and who is available to grade, and returns the evaluator. The second reads how many systems and what happens on failure, and returns the metric family plus a route to the live tool that computes it. Both are first-match rule tables: every rule is checked, the first that matches fires, and the rest are recorded as beaten or stopped.
Pass one: the evaluator
| Rule | When it fires | Evaluator |
|---|---|---|
| E1 | the oracle is a reference answer | exact-or-programmatic |
| E2 | the oracle is an acceptable set | exact-or-programmatic |
| E3 | the oracle is a rubric, and no human graders are available | llm-judge |
| E4 | the oracle is a rubric, and human graders are available on a sample | llm-judge |
| E5 | the oracle is a rubric, and human graders are available for every case | rubric-human |
| E6 | the oracle is a rubric, and a domain expert is available | human-expert |
| E7 | the oracle is expected tool calls | trajectory-or-state-check |
| E8 | the oracle is the expected final state | trajectory-or-state-check |
| E9 | the oracle is expected behavior only | human-expert |
Pass two: the metric
| Rule | When it fires | Metric | Route |
|---|---|---|---|
| M1 | two systems scored the same cases | Paired difference | paired-vs-independent-design-chooser |
| M2 | two systems scored different cases | Pass rate with an interval | independent-two-proportion-calculator |
| M3 | one system measured alone, and every attempt must succeed | pass^k | reliability-at-k-estimator |
| M4 | one system measured alone, and at least one attempt needs to succeed | pass@k | reliability-at-k-estimator |
| M5 | one system measured alone | Pass rate with an interval | pass-rate-ci-calculator |
Admissibility. The constraint matrix maps each oracle type to the evaluators it admits and the ones it rules out. Pairwise preference is inadmissible as an oracle-correctness evaluator for every oracle type here, because it measures which of two outputs a reader prefers, not whether either is correct against the oracle. Where selecting between candidates is the actual question, the pairwise win rate calculator's protocol chooser settles pointwise versus pairwise.
Validity evidence. Some evaluators need their own proof that the grading is sound. A model judge needs agreement against human labels on your own items, position-sensitivity, length-bias, and self-preference checks. A rubric-scored human evaluator needs inter-rater agreement corrected for chance. A domain expert needs at least two experts agreeing on a sample. An exact-or-programmatic evaluator needs none: the code is the definition.
Apart from the two passes, the grader stack (the "Which grader takes which cases" table) reads the oracle type, who can grade, how much of the criterion a code check can decide, and how the verdict is used. Five rules apply in order. (1) Expected tool calls or expected final state: code decides every case. (2) Expected behavior only: a domain expert grades every case. (3) A reference answer or acceptable set: code when it can decide all; a judge or person when it can decide none; code first, then the judge or person for the rest, when it can decide some. (4) A rubric: the same split, except that code for all is a conflict. (5) How the verdict is used adds notes, and for live monitoring it limits a person or expert to a sample of live traffic. Who can grade sets the grader for the remainder: with no human graders, a model judge that stays a candidate until a person labels a sample; with graders on a sample, a model judge checked by a person on a sample; with graders for every case, a person, checked by a second person; with a domain expert, the expert, checked by a second expert. The "What would change the grader" table counts changes to the code layer, the main grader or admissibility only.
| Task type | Metric | Grader | Pitfall |
|---|---|---|---|
| Classification or routing | Per-class precision and recall, macro-F1 | Code | Accuracy hides minority-class failure |
| Short-answer QA | Normalized exact match, token F1 | Code | A paraphrase fails exact match |
| Summarization | A validated judge's rubric pass rate; ROUGE-L as a smoke test only | Judge (validated) | ROUGE-L penalizes rewording that a person accepts |
| Translation | chrF, or BLEU with its settings stated | Code | BLEU is sensitive to tokenization and brevity |
| Retrieval | Recall at k, nDCG at k, MRR at the k the prompt uses | Code | A k larger than the prompt uses counts documents the model never sees |
| Agent task | End-state success, pass^k (succeeds every time in k tries), cost and steps per success | Code plus a judge or person for open-ended criteria | The agent may reach the goal by an unsafe path |
| Structured output | Schema-valid rate, field accuracy | Code | Output that passes the schema can still hold a wrong value |
| Safety | Harmful-compliance rate, over-refusal rate on a benign set | Code for format; a judge or person for substance | Over-refusal hides behind a low harmful-compliance rate |
| Metric | What it counts | Rewards | Misses |
|---|---|---|---|
| Exact match (after a stated normalization) | A score of 1 when the normalized strings are identical, 0 otherwise | Identical answers after normalization | Any correct answer written differently |
| Token F1 (bag of tokens, SQuAD normalization) | Harmonic mean of token-level precision and recall | Partial credit for overlapping content | Ignores word order; a wrong answer that shares many words with the reference scores well |
| BLEU (clipped n-gram precision to 4, brevity penalty) | Geometric mean of clipped n-gram precisions times a brevity penalty | Matching word sequences in the reference's order | Sensitive to tokenization; one reference gives noisy scores; the brevity penalty can dominate short outputs |
| ROUGE-L (longest common subsequence F1) | F1 of the longest common subsequence | The same content in roughly the same order | Penalizes valid rewording; misses factual errors |
| chrF (character n-gram F with beta = 2) | Character-level n-gram F-score, recall weighted twice as heavily | Matching character sequences, so inflected forms earn partial credit | Still a surface measure; misses semantic equivalence |
Pairwise preference measures which of two outputs a reader prefers, not whether either is correct against the oracle. It answers a different question. Where selecting between candidates is the actual question, the pairwise win rate calculator's protocol chooser is the right tool.
This happens when the oracle is expected behavior only and no human grader is available. The path forward is to turn the behavior description into a rubric, an acceptable set, or a set of expected calls, which opens the door to other evaluators.
pass^k, pass@k, and plain pass rate answer different reliability questions. pass^k (every attempt must succeed) falls with k; pass@k (at least one must succeed) rises with k. The retry model picks the one that matches what the product actually experiences.
The trace shows every rule the engine checked, which one fired, and where each non-firing rule stopped. It is the audit trail: if the recommendation looks wrong, the trace tells you which clause made it so.
No. It names the evaluator type and the metric family and links to the tool that runs the number. Each route in the recommendation block is a link to the next step.
Code checks are repeatable, fast and cheap to run. Their usual failure is a correct answer written in a form the check does not expect. Sending what code can handle to a judge wastes money and adds noise. Code takes the cases it can decide, and the judge or person takes only what remains.
Each row names an answer, or a group of answers to one question, and the grader that would result. A row appears when the code layer, the main grader or whether the grader can be used (a judge validated, an expert available) would change. How the verdict is used adds notes and, for live monitoring, limits a person to a sample; the grader stays as the other answers set it, so that question never forms a row.
For short-answer QA, use exact match or token F1. For translation, use chrF or BLEU with its tokenization and reference count stated. For summaries, use ROUGE-L only as a smoke test; a validated judge's rubric pass rate is the primary metric. The fold "What overlap scores reward and miss" lists what each metric rewards and what it misses.