Glossary
Design effect (grouped test outputs)
The design effect measures how much the variance of a pass rate grows when test outputs share a prompt family. With m checked outputs per family and within-family similarity rho, the design effect is 1 + (m-1) rho. Of n total outputs, only n divided by the design effect count as independent.
Sixty checked outputs, ten from each of six prompt families, with a within-family similarity of 0.2, have a design effect of 2.8 and carry the information of about 21 independent ones. A pass rate or an exit criterion computed as if there were 60 independent checks overstates the confidence: the interval is too narrow and the stated risk of passing a bad batch or failing a good one is too low.
Where m and rho come from
The m in the formula is the number of checked outputs per prompt family in the sample: one output from each family gives m equal to 1 and a design effect of 1. When families carry unequal counts, the right m is the sum of the squared counts divided by the total checked, which is larger than the simple average; enter that figure in the “Mean group size” field. The similarity rho is estimated from earlier runs with the same prompts; the worked example on the pass-rate page uses 0.15. Enter rho in “Within-group similarity.”
The pass-rate page’s “Grouped or clustered runs” panel takes those two numbers and widens the interval to match. The golden-set planner’s “Check a claim” mode sizes a set with the design effect applied.
A batch audit plan, where you check n outputs and accept the batch with at most c failures, carries a producer’s risk (rejecting a good batch) and a consumer’s risk (accepting a bad one), both computed from the independence assumption. Outputs checked several to a family carry less information than independent ones, so a plan that ignores the grouping understates both risks. The pass-rate page’s batch audit plan takes the sampling method (random across the batch or by whole families) as an input and sizes the plan for it.
Design effect and semantic duplicates
The semantic duplicate finder’s “Effective rows” drops rows that ask the same question in different words, removing actual repeats from the count. The design effect discounts distinct outputs that are merely alike because they share a prompt family. Duplicate removal shrinks the set; the design effect widens the interval on the set you keep. For suites with clustered prompts, a bootstrap with cluster resampling resamples whole families and accounts for the within-family similarity directly.