LatentEval

For builders

Agent evals: the four practices that change

Four practices that change when you evaluate an agent: grade the end state, report pass^k (succeeds every time in k tries), isolate trials and trace the first upstream failure.

By LatentEval Published Filed under Evaluation design

For builders

Agents change how each step of the evaluation loop is done, and four practices change when the system can act on the world.

Start here.

  • Before reading the agent’s final message, write the check that reads the end state after the trial: the record, the file or the database row.
  • Run each task k times from a fresh environment and report pass^k at the k the product depends on: reliability-at-k estimator.
  • On each failed trial, mark the first step that went wrong: error-analysis workbench.

Four practices change when the system is an agent

  1. Grade the end state the agent left. On AppWorld, false success (the agent reporting a task done that the end state shows it failed) was 75.8% of failures among the two of four architectures whose status field can report failure (the AppWorld arm holds 1,879 trajectories from four model families; the silent-failure study). In trace tr-rd-027 of the returns-desk agent traces, the label call times out at step 6 and the agent tells the customer a label was emailed: a grader reading the message passes the trial, and a check for the label fails it. Write the check that reads what must exist after the trial. The golden-set guides for transactional agents and coding agents give the spec.
  2. Report pass^k for reliability. pass^k measures reliability when the agent runs unattended (why one pass rate hides it). If trials fail independently, a per-trial pass rate of 0.9 gives pass^5 of 0.59. Set k to the number of consecutive successes the product depends on before anything checks the output (the checklist’s rule for k). The reliability-at-k estimator computes both pass^k and pass@k (succeeds at least once in k tries) from counts of passing and total trials.
  3. Isolate trials. Each trial starts from a fresh environment: a clean database, a reset file system, no leftover state from the previous trial. Without isolation, a label created in the first trial lets the second trial’s end-state check pass on work the agent never did. Trials that share state pass or fail together, which breaks the independence the pass^k arithmetic assumes (where the independence assumption bends). The golden-set guides in practice 1 give the reset step.
  4. Trace the first upstream failure. In 1,184 failed coding-agent trajectories on Terminal-Bench, the decisive error landed at a median of step 7 of a median 27-step trajectory, and the first observable signal came around step 16 (the trajectory study). Label each failed trial at the earliest step that went wrong; later steps inherit it. The error-analysis workbench records the first failing step; the agent trace analyzer maps the trajectory. Grading every step of a ten-step trajectory that went wrong at step two can record nine failures for one cause (one wrong step, nine failures).
Two-panel bar diagram: five 90-percent trial rows on the left, pass^5 of 0.59 on the right.
Five trials at a per-trial pass rate of 0.9 on the left; on the right, pass^5, the share that passes all five, is 0.59.

Each agent job has its own page

RAG and structured output need their own measures

For a RAG pipeline, the RAG retrieval metrics calculator scores retrieval, and the RAG evaluation research covers both stages, scored separately. For structured output, the structured-output failure rate and the golden-set guide for structured extraction cover schema failures and valid outputs with wrong values.