Run every eval on every pull request and teams stop running them

Your eval suite runs on every pull request, the same cases at the same trial count with the same wait every time. The check is either fast enough that it catches nothing, or thorough enough that your team has started overriding it.

One tier is doing three jobs. A pull request, a nightly build, and a release each ask a different question of the suite, and each one needs a different volume of evidence to answer.

A pull request, a nightly run, and a release each ask a different question

A pull request asks whether the change broke something obvious. A nightly run asks whether the suite is healthy on main. A release gate asks whether the specific regression the team agreed to block is absent.

The arithmetic behind the three tiers is already published. The sizing formulas, the test that says whether a nightly delta is real, and the detection each subset trades away are all in our own research: how many runs make an eval reliable, the agent reliability testing checklist, and what it takes for a delta to count as evidence. Catching a 10-point drop from a 90% baseline at 80% power takes roughly 70 distinct graded cases; catching a 2-point drop takes roughly 1,500. Blocking a 5-point regression at 80% power needs about 280 distinct cases per version. A per-PR delta that looks like a regression is evidence only when the subset is large enough to separate signal from noise.

This page assumes the distinction between observability and offline evals is already settled and the team has a suite. How long your eval suite adds to CI, and what a PR subset can detect covers the arithmetic of each tier: the time under parallelism, the monthly spend at a given cadence, and the detection each subset gives up. This page handles the placement: which checks run where, in what order, and what a failure at each tier should stop.

TABLEShow full table (3 rows)Showing full table (3 rows)

Tier	Blocks	When	What runs	Catches	Misses
PR smoke	The merge	Every push with an open PR	Deterministic checks, then a small pinned subset at one trial	Hard breakage: a prompt template that stopped rendering, a parser that broke, output that stopped being valid JSON. Large regressions	Anything inside the noise of a small subset. Slice-level regressions. Judge drift. Flakiness that needs repeats
Nightly full suite	Nothing. It opens an issue, marks the build, names an owner	On a schedule, against main	The whole set at the planned trial count on the frozen set version	Mid-size regressions. Regressions confined to one slice. Flakiness visible across the full set. Judge drift	Effects below the suite’s detectable drop. Anything needing many repeats per case
Release gate	The release	Before a promotion or release tag	The statistically sized run against a recorded baseline, with the gate rule written before the run	The smallest regression the team agreed to block, at the power they agreed to	Anything the suite does not contain. Anything the gate rule does not name

The Blocks column is what decides how the team reads a red build. A PR check that blocks nothing is a notification. A nightly run that blocks merges is a gate in the wrong place.

A fourth kind of rerun belongs outside the regular schedule. When a specific change ships, a model swap or a prompt edit or a retrieval change, only the suites and slices that change invalidates need to rerun. The change-impact worksheet maps a change to the suites it touches.

Deterministic checks run first because they fail fastest

Schema validation, format checks, forbidden-output patterns, tool-call contract enforcement. These cost no tokens. A prompt template that no longer renders, a JSON schema the model’s output stopped matching, a tool signature that changed: each one requires no model call and no network round trip, and catches breakage that would otherwise waste a full graded run.

The graded subset is the last thing the PR tier reaches. If the deterministic layer catches the fault, the model calls never fire.

When the suite has no judge at all and every check is deterministic, the PR tier and the full suite are the same check. The nightly tier becomes a rerun on main rather than a wider configuration.

The PR tier blocks a merge and the nightly tier files an issue

The asymmetry is deliberate. The PR tier blocks the merge. The nightly tier opens an issue and names an owner. The release tier blocks the release.

A nightly failure that blocks merges will turn into an override habit inside a week. Once overrides are routine, the tier has stopped being a gate and started being a delay. Record every override. If overrides are happening without records, the tier is either misplaced or the threshold is wrong.

The override path should exist. An engineer who needs to land a fix during an incident has a legitimate reason to override a nightly check. The rule is that each override is logged with the name, the reason, and the issue number it relates to. A pattern of overrides with no incident behind them is the signal that the tier needs to move.

In a monorepo where most pull requests touch nothing the suite covers, gate on path filters. A PR that changes a README shouldn’t wait for a model to answer. State explicitly what the path filter can miss, and accept that an occasional false negative is cheaper than a team that overrides every check.

If the team has no recorded baseline, there is nothing to gate against. Establish the baseline first; the release-gate designer writes the condition the release tier checks once a baseline exists.

A smaller subset catches less, and the team has to name what it gives up

A PR tier running the full suite catches every regression the suite contains. A PR tier running a fraction catches only what that fraction can see. The PR check prints its green badge and says nothing about the cases it did not run.

At 80% power, roughly 150 distinct graded cases catch a 5-point drop from a 95% baseline; from a 50% baseline, roughly 620 (how many runs for a reliable eval). The case count that gives the PR tier the detection the team wants, and the minutes that case count adds to the pipeline, is what the planner that sizes each tier computes. The classical sizing math behind it lives on the sample-size power calculator.

The release tier sits at the other end. Its case count comes from a power analysis, sized to catch the smallest regression the team agreed to block. The PR tier’s case count comes from what the team will wait for, with whatever detection that case count happens to provide. The difference between the two is the gap the nightly run fills, answering the full question every night without blocking anything.

When a suite is too slow for any PR tier at all, the right answer is a post-merge check on main. A thin gate pretending to be a regression check is worse than no gate, because it teaches the team that a green badge means the suite passed when it means a fraction of the suite passed.

A rerun that goes green tells you something about variance

A failing eval case that passes on rerun is evidence about the case’s variance. If a case fails one run in five, rerunning until it passes is equivalent to deleting it. The failure is data.

Pin the set version. Pin the seed and temperature where the harness allows it. And route the question of how many repeats a case needs before a single pass is genuine to the repeated-run variance planner. A pinned seed makes the same run reproducible. A failure that appears at one seed and disappears at another is a case that lives at the decision boundary, and pinning the seed lets the team debug it.

When the nightly run surfaces a delta the previous night did not show, the question is whether the delta is real. A one-night difference on a stochastic suite is evidence only when it survives a test. The McNemar test calculator takes the two counts and says whether the difference is noise or signal.

CI changes your eval in ways a local run does not

A suite that passes on a developer’s machine may fail on the CI runner. Three differences matter:

The set version has to be pinned, or the gate compares against a moving baseline. Pin the version and record the freeze date: a set that drifts between runs makes the baseline meaningless.

The judge is a dependency with its own version. A model upgrade on the judge changes every verdict, and the change has nothing to do with the system under test. Pin the judge model in the pipeline configuration, and treat a judge upgrade as its own change with its own rerun.

Rate limits and concurrency behave differently on a shared runner than on a laptop. A suite that runs eight requests in parallel locally may hit a provider rate limit on a runner shared across four pipelines.

The pipeline that runs the eval is an implementation detail; the placement of the tiers is the decision.

Four conditions that should never gate a release

A single-run delta with no repeat. One comparison on a stochastic suite, with no control for variance, is a measurement of noise. The nightly run surfaces it; the release gate or a follow-up rerun with the McNemar test is where it gets tested.

Cost as the sole gating signal. Whether the run is too expensive is a configuration question. Move the tier, shrink the subset, or change the cadence; the gate’s pass condition answers whether the system works.

A composite score. A number that blends accuracy, latency and coverage into one figure hides the dimension that moved. Report each metric separately and let the team see which one changed.

A threshold nobody wrote down before the run started. Choosing the bar after seeing the data is the same mistake however round the number is. The release-gate designer exists to write the condition before the run.

This page says nothing about what belongs in the eval set. If the suite measures the wrong thing, tiering the runs doesn’t fix the measurement. The placement starts with the table above. The CI planner computes what each tier configuration adds to the pipeline, and the release-gate designer writes the rule.