Golden set blueprint: summarization

Why this family’s correctness is different

A faithfulness number that does not say whether the error was a contradiction, an unsupported addition, or a dropped finding is a number that cannot point you at the fix. Intrinsic errors contradict the source; extrinsic errors add something the source never said; omissions drop something the source emphasized. The distinction only means anything once a source is named for each claim. And the evaluator granularity matters: a document-level consistency check misses errors a sentence-level check catches, so the granularity of the evaluator is itself a variable in the result.

This pack builds on research published on this site. The silent failure problem in AI evaluations documents how a single faithfulness number hides the failure shape. How LLM-as-a-judge evaluations work and where they break covers the granularity problem: a judge applied at the wrong level of analysis adds noise that looks like signal. A practical framework for AI agent evaluation establishes the general principle that each failure type needs its own denominator.

Case families

Six families. The first three (intrinsic-contradiction, extrinsic-addition, salient-omission) separate what a single faithfulness number collapses. The fourth (granularity-mismatch) catches the case where the summary does not match the requested detail level. The fifth (entity-confusion) catches the case where the summary attributes a fact to the wrong entity. The sixth (evaluator-granularity-failure) is a meta-case: it tests whether the chosen evaluator operates at a fine enough granularity to catch the errors the other families describe.

These counts seed a starter set; none of them supports a rate claim on its own. The coverage matrix floors set the claimable and readable thresholds.

A set built from this pack carries several oracle types across its families; run the evaluator navigator once per family and write acceptance criteria per family.

The evaluator-granularity-failure family is a meta-case. It does not test the summarizer; it tests the evaluator. A document-level faithfulness check that passes a summary containing a sentence-level contradiction has not measured faithfulness at the resolution that matters. Including these cases in the set means the set can detect when its own evaluation is too coarse.

Slice families

The input-length slice deserves special attention. Omissions are the error class most sensitive to source length: a system that omits nothing from a 500-word abstract may omit material findings from a 5,000-word report. A set built entirely on short sources has a structural blind spot.

Coverage matrix skeleton

Use the eval coverage matrix tool to fill the grid against your own system.

Evaluator and annotation guidance

Three evaluator configurations cover the six families. entity-confusion uses programmatic matching; four families use rubric-human; the evaluator-granularity-failure family uses an LLM judge (because the point is to test whether the judge catches what the human catches).

Reference-answer cases (exact-or-programmatic)

Used for entity-confusion, where the reference captures the correct attribution.

Validity evidence needed:

• The reference answer captures the correct attribution or value, not a paraphrase that would force partial matching.
• A sample of the passes and failures is read by a person to confirm the comparison is judging what you think it is.

Metrics: pass rate with a confidence interval, per-slice pass rate.

Tools: pass rate CI calculator, slice and class balance analyzer.

Rubric cases (rubric-human)

Used for intrinsic-contradiction, extrinsic-addition, salient-omission, and granularity-mismatch.

Validity evidence needed:

• Two raters on a sample, with their agreement reported and corrected for chance.
• The rubric operates at the sentence or claim level, not the document level, for intrinsic and extrinsic error detection.
• Disagreements resolved by rewriting the rubric, not by averaging.

The sentence-level requirement is load-bearing. A document-level rubric (“Is this summary faithful? Yes/No”) misses the errors this pack is designed to catch. The rubric must ask about each claim in the summary against the passage it should trace to.

Metrics: inter-rater agreement, pass rate, per-claim support.

Tools: inter-rater reliability calculator, multi-rater agreement calculator.

LLM-judge cases (rubric, llm-judge)

Used for the evaluator-granularity-failure family, where the point is to test the judge itself.

Validity evidence needed:

• Agreement with human labels on your own items, measured and reported with an interval.
• The judge operates at the sentence or claim level. A document-level judge adds noise that a claim-level judge avoids.
• The judge is calibrated against known contradictions, known additions, and known omissions before it grades unknown cases.

Metrics: agreement with human labels, calibration, per-claim support.

Tools: judge validation report builder, inter-rater reliability calculator.

Starter rows

These rows are starter examples showing what a case in this family looks like. They are not validated gold-standard answers. Before any of them grades a real output, someone who knows your system needs to confirm or rewrite the expected behavior in each one.

Each row’s input names the source document in square brackets rather than carrying it. The bracket is a specification of the document shape the case needs; substitute a document of your own that matches it before running the row.

Failure-to-case mapping

Use the risk-to-test mapper to trace from a named concern to the case families that test for it.

Downloads

Starter files for this pack, generated from the same source the examples above are read from: the CSV and the JSONL. Both carry the case profile in metadata.case. Every row is starting material, not a validated answer. This family’s files are published as summarization.csv and summarization.jsonl. To edit them in the browser and add your own, use the blueprint builder.

Which instruments apply

• Pass rate CI calculator for a confidence interval on the pass rate per family.
• Slice and class balance analyzer for checking whether the set covers each case-by-slice cell.
• Inter-rater reliability calculator for the agreement check on rubric-graded families.
• Multi-rater agreement calculator for agreement across more than two raters.
• Judge validation report builder for calibrating and validating the LLM judge against human labels.
• Eval coverage matrix for the grid of which cells are populated and which are empty.
• Golden set blueprint builder for editing and extending the starter rows.
• Risk-to-test mapper for mapping from a named concern to the case families that test for it.

Companion pack guides

• Structured extraction tests a related fidelity surface: whether the system transfers content from a source into a structured output without adding or dropping fields.
• Customer support covers multi-turn conversation, where the failure families include complaint recognition and escalation handoff alongside policy compliance.