{"id":"sum-001","input":"Summarize this quarterly earnings report in one paragraph. Source: [a 2,000-word report with revenue up 12%, net income down 3%, and a new product launch mentioned in the last paragraph].","expected":null,"split":"test","slice":[],"tags":["starter","single-source","executive-summary"],"metadata":{"case":{"caseId":"sum-001","caseFamily":"salient-omission","scenario":"A short summary of a financial report, where the last paragraph carries a material fact.","actor":"An analyst asking for a one-paragraph summary.","oracleType":"rubric","expectedBehavior":"Revenue, net income, and the new product launch all appear. None of the three is contradicted.","unacceptableBehavior":["Omits the new product launch from the last paragraph.","States net income rose when the source says it fell."],"evaluationCriteria":["Revenue figure is consistent with the source.","Net income direction is consistent with the source.","The new product launch is mentioned."],"reversibility":"R1","blastRadius":"B1","linkedConcerns":["omission","unsupported-claim"],"evaluator":"rubric-human","reviewerExpertise":"Someone who can compare the summary against the source report.","passLogic":"All three criteria met.","version":"1","retirementCondition":"The source report is replaced with a new one.","provenance":"starter-example"}}}
{"id":"sum-002","input":"Summarize this clinical trial abstract. Source: [a 500-word abstract reporting a primary endpoint met with p=0.03 and two serious adverse events in the treatment arm].","expected":null,"split":"test","slice":[],"tags":["starter","single-source","technical-summary"],"metadata":{"case":{"caseId":"sum-002","caseFamily":"intrinsic-contradiction","scenario":"A summary of a clinical abstract where both efficacy and safety findings must be present.","actor":"A researcher asking for a summary.","oracleType":"rubric","expectedBehavior":"The primary endpoint result, the p-value, and the two serious adverse events all appear. Neither the efficacy nor the safety finding is overstated.","unacceptableBehavior":["Omits the serious adverse events.","States the p-value as 0.003 instead of 0.03."],"evaluationCriteria":["The primary endpoint and p-value are stated correctly.","The adverse events are mentioned.","No overstatement of either efficacy or safety."],"reversibility":"R1","blastRadius":"B2","linkedConcerns":["omission","unsupported-claim"],"evaluator":"rubric-human","reviewerExpertise":"Someone who can read the abstract and verify numeric claims.","passLogic":"All three criteria met.","version":"1","retirementCondition":"The abstract is updated with new results.","provenance":"starter-example"}}}
{"id":"sum-003","input":"Give me a detailed summary of this contract. Source: [a 5,000-word services contract with a limitation of liability clause in section 12 and an auto-renewal clause in section 15].","expected":null,"split":"test","slice":[],"tags":["starter","single-source","detailed-summary"],"metadata":{"case":{"caseId":"sum-003","caseFamily":"extrinsic-addition","scenario":"A detailed summary where key clauses buried late in the document must appear.","actor":"A procurement manager asking for a contract summary.","oracleType":"rubric","expectedBehavior":"The limitation of liability and auto-renewal clauses both appear. The liability cap amount matches the source.","unacceptableBehavior":["Omits the auto-renewal clause.","States a liability cap different from the one in section 12."],"evaluationCriteria":["Limitation of liability clause is covered with the correct cap.","Auto-renewal clause is covered.","No fabricated clause appears."],"reversibility":"R1","blastRadius":"B1","linkedConcerns":["omission","unsupported-claim"],"evaluator":"rubric-human","reviewerExpertise":"Someone who can compare the summary against the contract.","passLogic":"All three criteria met.","version":"1","retirementCondition":"The contract template changes.","provenance":"starter-example"}}}
{"id":"sum-004","input":"Compare these two product specifications and summarize the differences. Source A: [spec sheet for model X]. Source B: [spec sheet for model Y].","expected":null,"split":"test","slice":[],"tags":["starter","multi-source","comparative"],"metadata":{"case":{"caseId":"sum-004","caseFamily":"entity-confusion","scenario":"A comparative summary where each difference must be attributed to the correct product.","actor":"An engineer comparing two products.","oracleType":"reference-answer","expectedBehavior":"Reference entity list: model X has feature A, model Y has feature B; model X weighs 2.1 kg, model Y weighs 1.8 kg; model X supports USB-C, model Y supports USB-A. Each stated difference names which product it belongs to.","unacceptableBehavior":["Attributes model X's specification to model Y.","States a difference that does not exist in either spec sheet."],"evaluationCriteria":["Every stated difference is real and correctly attributed.","No fabricated difference appears."],"reversibility":"R0","blastRadius":"B0","linkedConcerns":["wrong-entity","unsupported-claim"],"evaluator":"exact-or-programmatic","reviewerExpertise":"","passLogic":"Each entity-attribute pair checked against the reference list for correct attribution.","version":"1","retirementCondition":"Either spec sheet is revised.","provenance":"starter-example"}}}
{"id":"sum-005","input":"Summarize this meeting transcript in bullet points. Source: [a 3,000-word transcript where three action items are assigned to named people].","expected":null,"split":"test","slice":[],"tags":["starter","single-source","bullet-summary"],"metadata":{"case":{"caseId":"sum-005","caseFamily":"entity-confusion","scenario":"A bullet-point summary where action items and their owners must be preserved.","actor":"A project manager asking for a meeting summary.","oracleType":"reference-answer","expectedBehavior":"Reference entity-action pairs: Alice owns the vendor contract review, Bob owns the security audit follow-up, Carol owns the Q3 budget revision. Each action item appears with the correct owner.","unacceptableBehavior":["Assigns an action item to the wrong person.","Drops one of the three action items.","Returns prose instead of bullet points."],"evaluationCriteria":["Three action items listed.","Each action item has the correct owner.","Output is in bullet-point format."],"reversibility":"R0","blastRadius":"B1","linkedConcerns":["wrong-entity","omission","format-contract-violation"],"evaluator":"exact-or-programmatic","reviewerExpertise":"","passLogic":"Each action-owner pair checked against the reference list for correct attribution.","version":"1","retirementCondition":"The transcript is replaced.","provenance":"starter-example"}}}
{"id":"sum-006","input":"Source: [a 1,500-word news article about a merger where the acquiring company is X and the target is Y, with a stated price of $4.2B]. Summarize in two sentences.","expected":null,"split":"test","slice":[],"tags":["starter","single-source","entity-test"],"metadata":{"case":{"caseId":"sum-006","caseFamily":"entity-confusion","scenario":"A two-sentence summary where confusing the acquirer and the target reverses the meaning.","actor":"A reader asking for a concise summary.","oracleType":"reference-answer","expectedBehavior":"Reference entities: acquirer is X, target is Y, price is $4.2B. The summary must assign each role to the correct company and state the correct price.","unacceptableBehavior":["States Y is acquiring X.","States a price different from $4.2B."],"evaluationCriteria":["Acquirer and target are correctly identified.","Price matches the source."],"reversibility":"R1","blastRadius":"B1","linkedConcerns":["wrong-entity","unsupported-claim"],"evaluator":"exact-or-programmatic","reviewerExpertise":"","passLogic":"Each entity role and the price checked against the reference for correctness.","version":"1","retirementCondition":"The source article is replaced.","provenance":"starter-example"}}}
{"id":"sum-007","input":"Summarize this research paper, focusing on the methodology. Source: [a 4,000-word paper whose methodology section describes a randomized controlled trial with n=200 and three arms].","expected":null,"split":"test","slice":[],"tags":["starter","single-source","methodology-focus"],"metadata":{"case":{"caseId":"sum-007","caseFamily":"granularity-mismatch","scenario":"A methodology-focused summary where the design details must be accurate.","actor":"A researcher asking for a methodology summary.","oracleType":"rubric","expectedBehavior":"The study design (RCT), sample size (n=200), and number of arms (three) all appear correctly.","unacceptableBehavior":["States the wrong sample size.","Describes it as a two-arm trial when the source says three.","Adds a finding from the results section when the request said methodology."],"evaluationCriteria":["Study design correctly named.","Sample size correct.","Number of arms correct.","No results-section content leaks into the methodology summary."],"reversibility":"R0","blastRadius":"B0","linkedConcerns":["unsupported-claim","specification-violation"],"evaluator":"rubric-human","reviewerExpertise":"Someone who can read a methods section.","passLogic":"All four criteria met.","version":"1","retirementCondition":"The paper is retracted or updated.","provenance":"starter-example"}}}
{"id":"sum-008","input":"Summarize these three incident reports into one executive brief. Source A: [server outage, 45 min]. Source B: [data pipeline delay, 2 hours]. Source C: [minor UI bug, no downtime].","expected":null,"split":"test","slice":[],"tags":["starter","multi-source","executive-summary"],"metadata":{"case":{"caseId":"sum-008","caseFamily":"entity-confusion","scenario":"A multi-source summary where each incident must be correctly attributed and not merged.","actor":"An operations director asking for an executive brief.","oracleType":"reference-answer","expectedBehavior":"Reference incident-duration pairs: server outage lasted 45 min, data pipeline delay lasted 2 hours, UI bug caused no downtime. Each incident appears separately with its correct duration.","unacceptableBehavior":["Merges two incidents into one.","Attributes the 2-hour duration to the server outage."],"evaluationCriteria":["Three distinct incidents listed.","Each duration matches its incident.","No fabricated incident appears."],"reversibility":"R0","blastRadius":"B1","linkedConcerns":["wrong-entity","omission"],"evaluator":"exact-or-programmatic","reviewerExpertise":"","passLogic":"Each incident-duration pair checked against the reference for correct attribution.","version":"1","retirementCondition":"The incident reports are replaced.","provenance":"starter-example"}}}
{"id":"sum-009","input":"Summarize this safety data sheet. Source: [a 2,500-word SDS listing flash point 65C, acute toxicity category 4, and a first-aid instruction to induce vomiting only if directed by medical personnel].","expected":null,"split":"test","slice":[],"tags":["starter","single-source","safety-critical"],"metadata":{"case":{"caseId":"sum-009","caseFamily":"intrinsic-contradiction","scenario":"A safety-critical summary where a misquoted instruction could cause harm.","actor":"A warehouse supervisor asking for a summary.","oracleType":"rubric","expectedBehavior":"Flash point, toxicity category, and the first-aid instruction all appear verbatim or faithfully paraphrased. The conditional on the vomiting instruction is preserved.","unacceptableBehavior":["States the first-aid instruction without the medical personnel condition.","Changes the flash point value."],"evaluationCriteria":["Flash point is stated correctly.","Toxicity category is correct.","First-aid instruction preserves the conditional."],"reversibility":"R2","blastRadius":"B2","linkedConcerns":["omission","unsupported-claim","harmful-recommendation"],"evaluator":"rubric-human","reviewerExpertise":"Someone who can verify the safety data against the source SDS.","passLogic":"All three criteria met.","version":"1","retirementCondition":"The SDS is revised.","provenance":"starter-example"}}}
{"id":"sum-010","input":"Source: [a 6,000-word policy document with sections numbered 1 through 12]. Summarize section 8 only.","expected":null,"split":"test","slice":[],"tags":["starter","single-source","section-scoped"],"metadata":{"case":{"caseId":"sum-010","caseFamily":"granularity-mismatch","scenario":"A scoped summary where content from other sections must not appear.","actor":"A compliance analyst asking for a section summary.","oracleType":"rubric","expectedBehavior":"The summary covers section 8 only. No content from other sections appears unless section 8 explicitly cross-references it.","unacceptableBehavior":["Includes content from section 7 or section 9.","Omits a key provision in section 8."],"evaluationCriteria":["All key provisions of section 8 are covered.","No content from other sections appears unbidden."],"reversibility":"R0","blastRadius":"B0","linkedConcerns":["specification-violation","omission"],"evaluator":"rubric-human","reviewerExpertise":"","passLogic":"Both criteria met.","version":"1","retirementCondition":"The policy document is rewritten.","provenance":"starter-example"}}}
{"id":"sum-011","input":"Summarize this clinical trial report. Source: [a 3,500-word report where the primary endpoint was met (p=0.04) but one secondary endpoint showed a non-significant trend toward harm in a subgroup (p=0.12)]. A document-level faithfulness check says the summary is faithful.","expected":null,"split":"test","slice":[],"tags":["starter","single-source","evaluator-check"],"metadata":{"case":{"caseId":"sum-011","caseFamily":"evaluator-granularity-failure","scenario":"The summary omits the non-significant harm trend. A document-level NLI model rates the summary as faithful because none of its sentences contradict the source. A sentence-level check against the source would flag the missing subgroup finding.","actor":"A quality auditor checking whether the evaluator catches omissions at the right level of detail.","oracleType":"rubric","expectedBehavior":"The evaluator detects that the subgroup harm trend is absent from the summary. A document-level check that marks this as faithful is wrong, and the evaluation report says so.","unacceptableBehavior":["A document-level faithfulness check passes the summary without noting the omitted subgroup finding.","The evaluator reports high faithfulness because no sentence-level contradiction exists, ignoring the omission."],"evaluationCriteria":["The omission of the subgroup harm trend is flagged.","The evaluation distinguishes between contradiction-free and complete."],"reversibility":"R1","blastRadius":"B2","linkedConcerns":["verification-failure","omission"],"evaluator":"llm-judge","reviewerExpertise":"Someone who can compare the sentence-level and document-level verdicts and say which one is correct.","passLogic":"The judge flags the omission at the claim level. A document-level-only verdict that misses it is a fail.","version":"1","retirementCondition":"The source report or the seeded omission is changed.","provenance":"starter-example"}}}
{"id":"sum-012","input":"Summarize this financial report. Source: [a 2,800-word quarterly report where division A grew revenue 15% but division B declined 8%, and the CEO letter attributes the decline to a one-time restructuring charge]. A document-level check says the summary is consistent.","expected":null,"split":"test","slice":[],"tags":["starter","multi-source","evaluator-check"],"metadata":{"case":{"caseId":"sum-012","caseFamily":"evaluator-granularity-failure","scenario":"The summary says both divisions grew. A document-level consistency evaluator passes it because the overall company revenue did grow. A per-entity check against the source would catch that division B was misrepresented.","actor":"A quality auditor checking whether the evaluator catches entity-level errors hidden by a document-level average.","oracleType":"rubric","expectedBehavior":"The evaluator detects that the summary misrepresents division B as growing when the source says it declined. A document-level check that passes this is wrong.","unacceptableBehavior":["A document-level consistency check passes because overall revenue grew.","The evaluator treats the per-division error as immaterial because the aggregate direction is correct."],"evaluationCriteria":["The misrepresentation of division B is flagged.","The evaluation operates at the entity or claim level, not only at the document level."],"reversibility":"R1","blastRadius":"B1","linkedConcerns":["verification-failure","wrong-entity"],"evaluator":"llm-judge","reviewerExpertise":"Someone who can compare the sentence-level and document-level verdicts and say which one is correct.","passLogic":"The judge flags the per-division misrepresentation. A document-level-only verdict that misses it is a fail.","version":"1","retirementCondition":"The source report or the seeded error is changed.","provenance":"starter-example"}}}