Meta's four crawlers explained: ExternalAgent, WebIndexer, ExternalAds, ExternalFetcher

Meta documents more separate crawler tokens than most other operators, each for a distinct purpose. Writing one robots.txt rule intending to cover “Meta AI” usually misses at least one of them.

Which Meta crawler should I block?

To opt out of Meta’s AI model training while staying citable in Meta AI search, disallow Meta-ExternalAgent and leave Meta-WebIndexer allowed. Meta-ExternalFetcher fetches links a user asks for and, Meta says, may bypass robots.txt; Meta-ExternalAds serves Meta’s advertising products.

What does each Meta token do?

• Meta-ExternalAgent — training. Crawls the web “for use cases such as training foundation AI models or improving products by indexing content directly.”
• Meta-WebIndexer — search. Navigates the web to improve Meta AI search result quality.
• Meta-ExternalAds — advertising/validation. Crawls for use cases such as improving advertising and other business-related products.
• Meta-ExternalFetcher — agent. Fetches individual links at a user’s request, to support agentic AI capabilities in Meta products.

What happens when you block each one?

• Disallow Meta-ExternalAgent to opt out of Meta’s AI training use of your content.
• Disallow Meta-WebIndexer if you don’t want your pages surfaced in Meta AI’s search features.
• Meta-ExternalAds and Meta-ExternalFetcher serve narrower purposes (advertising products, and single-page fetches on a user’s request, respectively) — most sites leave these two allowed unless they have a specific reason to restrict them. For Meta-ExternalFetcher specifically, Meta’s own documentation says it “may bypass robots.txt because it performs fetches that were requested by the user,” so a Disallow for it is a declared preference, not a dependable control.

Does one rule cover “Meta AI”?

A robots.txt rule naming only Meta-ExternalAgent — the most widely discussed of the four — leaves Meta-WebIndexer, Meta-ExternalAds, and Meta-ExternalFetcher completely unaffected, since robots.txt groups never match on operator identity, only the exact token named. Note also that Meta’s own documentation writes these tokens in lowercase in the literal HTTP header (e.g. meta-externalagent/1.1) — matching is case-insensitive per RFC 9309, so either capitalisation works in a rule.

Check which of Meta’s four tokens your own robots.txt actually covers with the AI crawler checker.