Eval toolkits

Follow step-by-step lists for testing an AI feature, from the first question through live monitoring. Pick a role, or find your situation below.

Eval toolkits by role

Eval toolkit Product managers Set the quality bar, check the evidence and write the readout for a launch.
Eval toolkit Testers Read pass rates, find real changes in the noise and keep release evidence.
Eval toolkit Engineers Start from your own results files, audit them and wire evals into CI.

Guide
The eval loop for an AI feature

Each step of the evaluation loop, who does what and the page that does it.

By situation

Launch sign-off
Launch sign-off We launch next week and I have to sign off, or I was sent a readout to approve
- Write the gate rule
- Write the eval readout for a launch decision
First evals from logs
First evals from logs We have logs or a feedback spreadsheet, but no evals yet
- Your first error-analysis session
- Criteria and rubric builder for LLM eval grading
Green metrics, unhappy users
Green metrics, unhappy users Metrics look fine, but something is wrong
- Prove your eval suite can fail
- Error analysis workbench for LLM traces and eval results
Model or vendor change
Model or vendor change Our model is being retired, it changed under us, or we are choosing a new one
Leaving an eval platform
Leaving an eval platform We are moving off an eval platform and need to bring our data with us
- Convert the file to another format
- Test whether the harness moved your score
Customer questionnaire
Customer questionnaire A customer sent us an AI questionnaire
- Write the eval readout for a launch decision
Live monitoring
Live monitoring We are live: what do we watch, and is a 2% thumbs-down rate good?
- Thumbs-down rate: what does 2% really mean?
- Review sample drawer: random and stratified sampling for eval review
Prompt change or a small drop
Prompt change or a small drop We changed the prompt, or the score dropped 2 points: did anything break?
Flaky CI tests
Flaky CI tests Our AI tests fail at random in CI and the team has stopped trusting them
Inherited test suite
Inherited test suite I just inherited these tests and have to vouch for them
"How much is enough" is a sample-size question because no black-box coverage criterion exists for a model; find uncovered requirements in the coverage matrix and risks in the risk-to-test mapper.
Escaped defect in an agent
Escaped defect in an agent A defect escaped and I need to write the case and show it fails on the old build
- A bug you already fixed should show as fixed
- First recorded errors: group the failed runs