Testers
Listed in the order of a tester's week, from a red test to release.
An LLM feature can answer differently from one run to the next, so a test of it has a pass rate: most steps below read a rate, compare rates or write one down. The AI field calls this kind of test an eval.
Testing an LLM feature walks the eight steps, each with its page and example.
Other toolkits Product managersEngineers
Setting up evals
- A CI test failed and a single run does not settle it Reruns of one failing case: break or noise?
- Retried to green: how often would it pass without one? Chance that every run passes
Also opens CI retries: what a green build hid
- Nothing failed: what pass rate can the release claim? Runs for a claim: what a clean result proves
- Two builds, same cases: which broke and which were fixed? McNemar test for paired runs
- Something changed: which tests to rerun, from what baseline? Change impact worksheet
- A model judge makes mistakes that affect the pass rate Corrected rate: account for the judge's error
In production
This toolkit does not cover security red-teaming and prompt injection, load testing, bias and fairness audits, observability dashboards or generated test suites.