LatentEval

Product managers

Listed in the order the work comes up across a feature's life.

Other toolkits TestersEngineers

Setting up evals

  1. A new feature, and no plan for how to test it How much evaluation does this feature need?
  2. Write down what good looks like Criteria and rubric builder for LLM eval grading
  3. Look at what went wrong, case by case Error analysis workbench for LLM traces and eval results

    Also opens Your first error-analysis session

  4. No quality bar yet, and results are about to come in Set the quality bar before the run
  5. How sure are we, and is the grading reliable? Is this eval number ready to quote?
  6. Write the readout Write the eval readout for a launch decision

In production

  1. A model change is under way When the model changes: the playbook
  2. Once it is live: what to watch Build a monitoring plan for your AI feature

Next step