Golden Set Blueprint Builder: Seed a Test Set From a Task Family

Support correctness is a property of the conversation, not of any single answer. The binding constraint typically appears several turns before the response that must honor it, so a set of isolated single-turn questions evaluates a system nobody actually runs. Three capabilities here are testable and routinely omitted: whether a message satisfying the written complaint definition is recognized as one, whether the system hands off when the situation demands a person, and whether it declines requests it should have handled. The last two move in tandem, so measuring only one tells you nothing about the cost of the other.

Read the Customer support AI blueprint guide (built on how agent errors go undetected, trajectory-level agent evaluation )

A single hallucination rate collapses seven distinct failure points into one number nobody can act on: the answer was not in the corpus, the passage was not retrieved, the passage was retrieved and ignored, the source was stale, the synthesis was incomplete, the specificity was wrong, or the format was wrong. Score retrieval and generation separately with an interval on each. A perfect retrieval score is compatible with an answer that ignored the passage, so a generation check exists whether or not retrieval was measured.

Read the Enterprise knowledge and RAG assistant blueprint guide (built on RAG pipeline failure modes, evaluating a RAG pipeline beyond a single score, undetected failures in agent output )

Extraction correctness splits into five classes that per-field F1 collapses: the field is missing (omission), the field is fabricated (hallucination), the span boundary is wrong, the value lands in the wrong field (type or label error), and the output is schema-valid but factually wrong. Per-record and per-field metrics diverge because a record with nine correct fields and one hallucinated one is 90 percent right per field and wrong per record. Report both, with the denominators stated.

Read the Structured extraction blueprint guide (built on structured output failures across models, false-success failures )

A faithfulness number that does not say whether the error was a contradiction, an unsupported addition, or a dropped finding is a number that cannot point you at the fix. Intrinsic errors contradict the source; extrinsic errors add something the source never said; omissions drop something the source emphasized. The distinction only means anything once a source is named for each claim. And the evaluator granularity matters: a document-level consistency check misses errors a sentence-level check catches, so the granularity of the evaluator is itself a variable in the result.

Read the Summarization blueprint guide (built on agents that succeed but are wrong, testing LLM-as-a-judge reliability, following the full agent trajectory )

The oracle here is world state, not text. Grading the message the agent wrote about what it did measures the message, not the action. A correct answer beside an incorrect write, or two writes when the request called for one, passes every text grader and fails every state check, which is why they are different instruments. Three properties are first-class case classes in this family and are usually footnotes elsewhere: whether confirmation is re-sought when the details change mid-conversation, whether the agent stops honestly when it runs out of budget, and whether a tool error is handled rather than swallowed.

Read the Transactional tool-using agents blueprint guide (built on why agent errors look like successes, evaluating the complete agent trajectory )

The test suite is the evaluator, not the output text. An unaudited test suite is an unvalidated evaluator: a test that asserts the current output rather than the specified behavior passes everything and validates nothing. The decisive mistake lands in the first file the agent edits and stays invisible for many subsequent steps, which is why trajectory matters alongside final state. Solution leakage inside the issue text inflates success without inflating capability, and the gap between a leaked and an unleaked variant of the same bug is the inflation estimate.

Read the Coding agents blueprint guide (built on how coding agents fail in practice, evaluating agents across the full trajectory )

Correctness here is per-claim, not per-response. Each claim must be checked against a named source, and citation existence, citation support, and citation strength are three separate failure classes. An agent that cites a real source, quotes it faithfully, and overstates its strength fails on the third without failing on the first two. The headline risk is silent success: the output reads as confident and well-sourced while the claims are unsupported or fabricated. A response-level pass rate hides this because the response looks good.

Read the Research agents blueprint guide (built on when agents report success incorrectly, whole-trajectory evaluation )

Content generated in a setting where rules govern what may and may not be said is graded on must-not-do constraints checked per output, not on a quality or fluency score. A missing risk disclosure is a different failure from an inaccurate claim. An audience mismatch is a different failure from a claim that cannot be traced to its source. And evidence-strength mischaracterization is a different failure from all three. Each of these is counted with its own denominator rather than folded into a single accuracy rate, because a single rate hides which constraint was broken.

Read the Regulated and controlled content blueprint guide (built on silent failures in agent systems, how benchmarks get gamed )