Glossary
Test oracle (LLM features)
A test oracle is the source of the expected result a test compares against. On an LLM feature, a grader (a code check, a person or a model judge) applies the oracle, and every grader has its own catch rate and false alarm rate that move the observed pass rate.
Every test compares the system’s output against an expected result. That expected result is the oracle. On an LLM feature the oracle can be a reference answer, a set of acceptable answers or a rubric, and a grader applies it: a code check, a person reviewing a sample, or a model judge.
The grader’s own errors move the observed pass rate. With a true pass rate of 80%, a grader that catches 85% of real failures and wrongly fails 10% of real passes produces an observed rate of 75%. The observed pass rate mixes the system’s true rate with the grader’s errors, and the true rate cannot be recovered from it until the grader’s two rates are known.
Carrying the grader’s error into the rate
Once you have the grader’s catch rate and false alarm rate, each measured on outputs a person marked as failures and on outputs a person marked as passes, you can back out the true pass rate from the observed one. The bias correction page does that arithmetic: enter the labeled counts, the number judged to pass and the total tested, and it returns the corrected rate.
Labeling enough outputs to trust those rates is judge validation: plan the sample and record the result so the rates travel with the grader. When the grader changes, re-check it; a new model version or a changed rubric is a new grader.
What a test oracle is not
A task verification failure is what happens when a checking step inside the system under test passes wrong work along. Both that step and an external oracle can let a failure through, but one sits inside the system and the other outside it.
Judge calibration covers whether a judge’s stated confidence matches how often it is right. Agreement with human labels, measured separately on real failures and real passes, is a different measurement.
The oracle types available for an LLM feature, and which grader may apply each, are described in the case schema guide.
Where no expected result can be written, a reference version or a property that must hold on every output can serve as the oracle. Back-to-back testing uses the reference version: run both versions on the same inputs and check the differences against rerun noise.