Prompt Robustness Analyzer: Perturbation Stability for LLM Evals

The paired design. Every item is scored once under the baseline prompt and once under each perturbation variant, so each variant forms a paired 2x2 table with the baseline on the items that carry both results: both pass, pass to fail, fail to pass, both fail. Pairing subtracts each item's own difficulty out of the comparison, which is what lets a small suite say anything about a perturbation at all. Items are the independent unit; the variants are repeated conditions on the same items, so nothing here treats variant cells as independent draws.

Degradation. The pass-rate change is the variant pass rate minus the baseline pass rate on the paired items, which reduces to fail-to-pass minus pass-to-fail over n. Its interval is Newcombe's (1998) method-10 paired-difference interval: the two Wilson score intervals squared and added through the paired phi correlation, with the continuity correction on phi that Newcombe recommends (a positive ad minus bc is reduced by n over 2 and floored at zero). The correction matters most where this page lands often: a variant with no flips at all has a raw phi of 1 and would otherwise carry an unrealistically tight band on a small suite. The interval behaves at small counts where a Wald interval would not; the same paired interval on a single delta is what the paired test for one variant reports. The exact McNemar p is the two-sided binomial test on the smaller flip count against a fair coin over all flips: it asks whether the net change is more than paired noise. With several variants the p-values are unadjusted; the page also states the Bonferroni bar of alpha over the number of variants for a reader who wants a family-wise answer.

Flips and stability. The flip rate is both flip counts over paired items, with a Wilson interval, and is kept apart from the change on purpose: pass-to-fail and fail-to-pass cancel in the change but add in the flip rate, so a variant can churn results without moving the pass rate. Per-item stability is the share of that item's present variants whose outcome agrees with the baseline; an item is fully stable when every present variant agrees. The headline pools flips over every paired cell. The share of fully stable items carries a Wilson interval because items are the independent unit; the pooled flip rate does not, since the same item contributes one cell per variant.

Robust pass rate and spread. Passing under the baseline and every variant is the perturbation counterpart of pass^k in the reliability@k estimator: the item passes under all conditions, so a single-phrasing pass rate that looks fine can sit well above it. Passing under at least one condition is the counterpart of pass@k. The spread is the best condition pass rate minus the worst, the figure the site's guide on prompt wording as an unswept hyperparameter asks teams to report beside the pass rate; the baseline is measured over every item that has a baseline result and each variant over its own paired items.

Missing cells. An item that lacks a result under one variant drops out of that variant's paired table and out of that item's stability denominator; a row with no baseline result is excluded from every paired figure. Both are counted in the summary so nothing disappears silently. Any malformed line rejects the whole paste rather than dropping rows.

Worked example. The preloaded 20 items pass at 75% under the baseline. The format variant drops the pass rate to 35%, a change of -40.0 points with a 95% interval of -58.3 to -14.3 points that clears zero and an exact p of 0.008, which also clears the Bonferroni bar of 1.25% for four variants. The reorder variant flips 2 of 20 results but nets a 0-point change, so it reads as unstable in the flip table and as harmless in the change column; both readings are correct answers to different questions. Pooled over 80 paired cells the flip rate is 20.0%, half the items hold their baseline result under every variant, and only 35% of items pass under every condition against the 75% the single-phrasing number reported.

Honest limits. The variants say only as much as the perturbations you chose to run: a suite that never varied the option order will never show order sensitivity. Stability is agreement with the baseline, and a stable wrong answer is still wrong, so read the change column beside it. Each condition is one run, so run-to-run noise inside a condition is invisible here and adds to what a flip rate shows; the piece on measuring agent reliability past a single pass rate covers that variance. On small suites one flip moves a variant's rate by a whole item and the intervals will say so; the tool states the item count beside every figure for that reason. A rewording that moves the rate this much belongs in the change-impact worksheet, not only in this run's own report.

Formula: change = (fail to pass minus pass to fail) / n; flip rate = (pass to fail + fail to pass) / n; stability = agreeing variants / variants present

See Sources below