Glossary
Simulated user (agent tests)
A simulated user is a model that plays the user's role in a multi-turn agent test, with its own error rate measured on a reviewed sample of conversations the same way a model judge is checked against human labels.
In the tau2-bench paper (arXiv 2506.07982, version 1), Barres et al. generated traces with OpenAI’s GPT-4.1 serving as both the simulated user and the agent, and two annotators reviewed each conversation: 50 conversations in retail and 100 in airline. The tau2-bench user-simulator error rates per conversation reviewed were 40% in retail and 47% in airline, with 12% and 13% critical errors (those that blocked task completion). The paper counts errors and does not say whether one conversation can hold more than one. An agent failure in a conversation that also contains a simulator error may be the simulator’s. Until you measure the simulator’s error rate on your own task with your own setup, any agent failure rate you report carries the simulator’s errors inside it, and you cannot tell how much of the rate belongs to each.
tau2-bench has had later releases, and its maintainers state that scores from release 2.0.0 are not comparable with 1.x releases. The simulator error rates cited above come from the paper’s own experimental version and belong to that setup. They are error rates from a human review of generated conversation traces. They are not benchmark scores.
Measuring the simulator’s error rate
Review a sample of conversations from your test runs. Load them into the error analysis workbench, code each simulator error and whether it is critical (blocking task completion), and count the conversations by code. Check the simulator error share with an interval on the pass-rate page: enter conversations without a simulator error as “Passes (c)” and total conversations reviewed as “Total runs (n).” The error share is one minus the pass share, and the interval bounds flip too (one minus the upper pass bound is the lower error bound).
Then decide what to do with conversations that contain a critical simulator error: drop them, run them again with a different simulator setup, or report the agent’s pass rate with the simulator’s error rate alongside it. The habit is the same as checking any test oracle: a simulated user has an error rate that enters the measurement, and a human review of a sample is how you measure it.