CrawlPactMeta · Search crawler
Meta-WebIndexer
Navigates the web to improve Meta AI search result quality.
Questions about Meta-WebIndexer
What is Meta-WebIndexer?
Meta says it navigates the web to improve Meta AI search result quality, and that allowing it helps Meta cite and link to your content in Meta AI's responses.
Is Meta-WebIndexer used for AI training?
Meta describes training as a use of Meta-ExternalAgent, a separate crawler with its own robots.txt token.
Meta documents Meta-WebIndexer as a crawler that “navigates the web to improve Meta AI search
result quality for users” by analysing content to enhance relevance and accuracy — a
search-purpose crawler, distinct from Meta-ExternalAgent’s training purpose.
A note on the token
Meta’s own documentation writes the literal user-agent string in lowercase,
meta-webindexer/1.1. robots.txt user-agent matching is case-insensitive (RFC 9309 §2.2.1), so
this does not change how a Disallow rule applies — CrawlPact displays crawler names in the
capitalisation convention used consistently across this directory.
Site-owner controls
Disallowing Meta-WebIndexer affects this content’s relevance and accuracy in Meta AI’s search
results specifically. It does not affect Meta-ExternalAgent (AI training), Meta-ExternalAds
(advertising validation), or Meta-ExternalFetcher (user-triggered agent fetches) — each is
governed by its own separate token. See /limitations for what a robots.txt rule
can and cannot guarantee.
Example robots.txt configuration
To disallow Meta-WebIndexer specifically, without affecting any other crawler:
User-agent: Meta-WebIndexer
Disallow: /If no dedicated User-agent: Meta-WebIndexer group exists in a domain'srobots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works. A robots.txt rule is a published instruction; it does not technically prevent access.
Official source
https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
Verified against the source above as of 24 July 2026 — see how CrawlPact verifies crawler information.
Related guides
- Blocking AI training while staying visible in AI searchChoosing between CrawlPact's presets when the goal is opting out of model training without losing AI-search discoverability.
- Meta's four crawlers explained: ExternalAgent, WebIndexer, ExternalAds, ExternalFetcherMeta documents four separate crawler tokens with four separate purposes. A decision guide for telling them apart before writing a robots.txt rule.
Other Meta crawlers
- Meta-ExternalAdsCrawls the web for use cases such as improving advertising and other business-related products.
- Meta-ExternalAgentUsed by Meta to crawl content for training AI models and improving AI products.
- Meta-ExternalFetcherFetches individual links at a user's request to support agentic AI capabilities in Meta products.
Other search crawlers
See how this applies to your own site
Run a free audit to check your declared AI crawler policy against your own domain, or use the AI crawler checker to see how your robots.txt treats Meta-WebIndexer and every other documented crawler.