TOOL | Evaluation design
Agent Trace Analyzer: Step Counts, Repeated Calls, Handoff Edges
Upload a trace file from a finished agent run. The page counts steps, tool calls, repeated-call runs with index ranges, and handoff edges, built on the step-level failure patterns documented in the trajectory failure analysis. It reads OpenTelemetry GenAI spans, the OpenAI Agents SDK export, and plain tool-use message logs.
The run-class counts link to the pass-rate interval calculator for the rate and its interval. To record a run rather than read one, use the eval run register. To set an iteration cap from many runs' step counts, use the agent loop cap calculator.
Upload or paste a trace file. In "First recorded errors" mode, also supply a results file. The page counts steps, tool calls, repeated-call runs, handoff edges, per-class run tallies and, with results, first recorded errors. It reports what the files record and nothing else.
- Format
- Ordering rule
- Runs in file
Run classes
| Measure | Value |
|---|
To put an interval on a class rate, take the count to the pass-rate interval calculator.
First recorded errors
These counts are from a fictional returns-desk agent team with planted faults. Change either file and this caption disappears.
Join counts
| Measure | Value |
|---|
Groups by first recorded error
| Agent | Step kind | Step name | Failed runs | Cases | Passing runs, same first recorded error | Member runs |
|---|
Groups by first recorded error
| Agent | Failed runs | Cases | Passing runs, same first recorded error | Member runs |
|---|
Failed runs and first recorded errors
| Run | Case | First recorded error index | Error steps after | Group |
|---|
Failed runs with no recorded error
| Run | Case |
|---|
The groups above are a starting list, not a count of defects. A study of coding-agent failures found the first visible sign of trouble can appear many steps after the decisive step; on this page the first recorded error can sit as far, and it can be an unrelated earlier error already resolved. One defect can spread across groups and several defects can share one. Confirming that runs share a defect means rerunning them with the earlier step's output corrected. Failed runs with no recorded error need reading in full.
Warnings
The shared link carries your format, thresholds, grouping choice and mode. The trace file, the results file and the handoff contract stay in your browser.
| Measure | Value |
|---|
Repeated-call checks
| Check | Measurement | Over your threshold |
|---|
Handoff edges
| From | To | Handoffs |
|---|
Contract check
| Required item | Handoffs missing it |
|---|
How this is calculated
Steps are ordered by timestamp when every step has one. When some timestamps are missing, or the format provides none, steps stay in file order for the entire run. The rule used is shown in the header.
A step's signature is the combination of its kind, its tool name and its canonical argument serialization. Two calls to the same tool with the same arguments produce the same signature. Three checks run over the signature sequence of each run:
- Immediate repeat (L1). The longest consecutive run of one signature. Flags when the run length reaches the repeat threshold you set.
- Cycle repeat (L2). The longest consecutive repetition of a multi-step pattern. For each window length from 2 up to the max cycle length, the page finds the maximum number of consecutive repetitions. Windows whose steps are all the same signature are skipped so L1 runs are not double-reported. Flags when a pattern repeats at least twice after its first appearance.
- No-new-signature window (L3). The longest run of consecutive steps that use only signatures the run has already seen. A new tool call resets the window. Flags when the window length reaches the threshold you set.
Each run is classified into exactly one class. The first rule that matches wins:
| Class | Rule |
|---|---|
| Unknown | The format carried no status signal anywhere in the run: no step has a recorded status, the root span has no status, and the last step has no output. |
| Errored | Any step has a recorded error status, or the root span status is error. |
| No output | Not errored, and the last step produced no output. |
| Completed | A status signal exists, nothing errored, and the last step produced output. |
One trace is a sample of one. To put an interval on a class rate measured across runs, use the repeated-run variance planner to decide how many runs the interval needs.
The mode joins the trace file to the results file on a trace id column (trace_id, traceId or Trace ID). When none exists, it uses a run_id column whose values match runs in the trace file; when neither exists, it refuses. Each run's verdict comes from the overall scorer row when one exists; otherwise any failing row marks the run as failed, and all-blank rows leave it with no verdict.
On this page, an error is a step the trace records with an error status; the word marks a status, not a person's mistake. The first recorded error is the first error step in the order the page shows, with no errored step beneath it in the parent chain. An agent span that errors because a child errored is a wrapper; the mode looks past it. This can differ from the run panel's first error index, which ignores nesting. The count after it is later error steps, wrappers left out.
Groups share the same agent, step kind and step name at the first recorded error. The agent is the nearest enclosing agent in the parent chain. The step kind is the type: tool, agent, model call, handoff, message or other. The step name is the tool name, agent name or model. Groups are ordered by failed-run count, ties broken by file position.
The mode counts first recorded errors and groups the failed runs that share one. It reads no message text, makes no model call, computes no rate and assigns no failure to any agent or step. The rule that finds each run's first recorded error and groups by it is our synthesis.
Questions
Which trace formats are accepted?
OpenTelemetry GenAI spans exported as OTLP/JSON, the OpenAI Agents SDK span export, and plain tool-use message JSONL. The format is detected from the file. If detection is wrong, override it with the format selector.
What does "not applicable" mean for handoffs?
The plain message JSONL format does not carry handoff information. When the file uses that format, handoff counts are reported as not applicable rather than as zero, because zero would imply the page looked and found none.
Why does the page not say what caused a failure?
The page counts what the trace records. It does not interpret, diagnose or attribute a cause. A step that errored is counted as errored. What went wrong is a question for the trace itself, not for a counter reading it.
What is the handoff contract field for?
It lists what the receiver of a handoff was supposed to get. Each line is a literal string or a regex pattern. The page checks each handoff's inbound payload against every item in the contract and reports which items were missing. Without a contract, the page still counts handoff edges, but the loss measurement has nothing to measure against.
Does one group mean one defect?
No. One defect can spread across several groups when it produces different first recorded errors on different runs. Several defects can share one group when they happen to fail at the same step. The groups are a starting list: a person confirms whether runs in a group share a defect by rerunning them with the earlier step's output corrected, or by reading the traces on the error analysis workbench. The failure attribution entry covers pinning a failure to one agent and step.
Why does this mode need the results file?
A run the grader failed can record no error at all, and a run that recorded an error can still pass. In the returns-desk example, four failed runs show no recorded error, and two passing runs show one. The trace alone cannot tell which runs failed. The results file supplies the verdict, and the mode counts only the failed runs' first recorded errors.