Glossary
Back-to-back testing (LLM features)
Back-to-back testing runs a reference version and the version under test on the same inputs and compares their outputs, using the reference as a pseudo-oracle. On LLM features, each disagreement is checked against how often the same cases flip on a plain rerun.
The ISTQB AI testing syllabus (CT-AI v2.0), section 6.1.10, describes back-to-back testing as a technique where an alternative version serves as a pseudo-oracle. The test needs only inputs, because the reference version’s outputs stand in as the expected results. The technique suits migrations and environment moves, and on an LLM feature a model version swap is a typical use.
Both versions vary run to run, so some disagreements are noise from reruns. Comparing the differences paired on the same test cases, each against how often the same cases flip on a plain rerun, is our synthesis. Research on whether an eval difference is statistically significant gives a paired test on the cases the two versions disagree on, asking whether more cases flip one way than the other.
If both versions share a component, the same base model or the same grader, they can agree on a case while both produce wrong output. Shared agreement tells you only that the two versions behave alike. Whether either version is correct needs an oracle independent of both, like human labels on a sample. Those labels settle which version meets the bar and give its catch rate and false alarm rate.
Reading the differences
A single paired comparison gives both confirmation (cases that failed before and pass after) and regression (cases that passed before and fail after) in one pass, each direction counted on the same cases. Open the two result files, one per version, in the A/B comparator’s “Same items, joined on id” mode to see which cases changed direction and by how many. The McNemar test asks whether the number of cases fixed and the number broken differ by more than chance. To put the version difference in context, open a single version’s reruns in the repeated-run variance planner’s “Run rows” mode: that rerun baseline shows how far the same cases flip on a plain rerun of one version alone.
In CT-AI’s usage, A/B testing compares variants using performance metrics and statistical techniques, while back-to-back testing aims at finding defects. Online A/B traffic splits, which route live users to different versions and measure conversion or engagement, are a different practice.