CrawlPactMeta · Training crawler
Meta-ExternalAgent
Used by Meta to crawl content for training AI models and improving AI products.
Questions about Meta-ExternalAgent
What is Meta-ExternalAgent?
Meta says this crawler is used for use cases such as training foundation AI models or improving products by indexing content directly.
Is it the same as Meta's AI search crawler?
No. Meta documents Meta-WebIndexer separately for improving Meta AI search results, and Meta-ExternalFetcher for links a user asks it to fetch — which Meta says may bypass robots.txt.
Meta-ExternalAgent is documented by Meta as a crawler used to gather content for training AI
models and for improving Meta’s AI products and features.
Site-owner controls
Disallowing Meta-ExternalAgent affects only Meta’s AI-training use of this content. It does not
affect Meta-WebIndexer (search), Meta-ExternalAds (advertising validation), or
Meta-ExternalFetcher (user-triggered agent fetches) — Meta documents each as its own distinct
token, and CrawlPact evaluates them independently. See /limitations for what a
robots.txt rule can and cannot guarantee.
Distinguishing from Meta’s other crawlers
Meta operates several distinctly named crawlers for different purposes (link previews,
indexing, and AI training). CrawlPact’s registry tracks Meta-ExternalAgent specifically as a
training-purpose crawler; other Meta tokens are evaluated separately as the registry expands.
Example robots.txt configuration
To disallow Meta-ExternalAgent specifically, without affecting any other crawler:
User-agent: Meta-ExternalAgent
Disallow: /If no dedicated User-agent: Meta-ExternalAgent group exists in a domain'srobots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works. A robots.txt rule is a published instruction; it does not technically prevent access.
Official source
https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
Verified against the source above as of 1 July 2026 — see how CrawlPact verifies crawler information.
Related guides
- Blocking AI training while staying visible in AI searchChoosing between CrawlPact's presets when the goal is opting out of model training without losing AI-search discoverability.
- How to block only AI training crawlers, without blocking AI searchA step-by-step implementation guide for writing robots.txt groups that restrict training-purpose crawlers while leaving search-purpose crawlers untouched.
- Meta's four crawlers explained: ExternalAgent, WebIndexer, ExternalAds, ExternalFetcherMeta documents four separate crawler tokens with four separate purposes. A decision guide for telling them apart before writing a robots.txt rule.
Other Meta crawlers
- Meta-ExternalAdsCrawls the web for use cases such as improving advertising and other business-related products.
- Meta-ExternalFetcherFetches individual links at a user's request to support agentic AI capabilities in Meta products.
- Meta-WebIndexerNavigates the web to improve Meta AI search result quality.
Other training crawlers
See how this applies to your own site
Run a free audit to check your declared AI crawler policy against your own domain, or use the AI crawler checker to see how your robots.txt treats Meta-ExternalAgent and every other documented crawler.