CrawlPactGoogle · Training crawler
Google-Extended
Controls use of website content for training Gemini and Vertex AI generative models, independent of Search indexing.
Questions about Google-Extended
What is Google-Extended?
A robots.txt token that controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it has no separate HTTP user agent; crawling uses Google's existing user agents.
Does blocking Google-Extended affect Google Search?
No. Google says Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal.
Does Google-Extended control AI Overviews or AI Mode?
No. Google names robots.txt rules for Googlebot, plus the nosnippet, data-nosnippet, max-snippet and noindex controls, as the controls for AI features in Google Search.
Google-Extended is a control token, documented by Google, that lets a website opt out of
having its content used to train Gemini and Vertex AI generative APIs — without affecting
Google Search indexing, which is governed separately by Googlebot.
Why this one is easy to get wrong
Disallowing Googlebot blocks Search indexing entirely. Disallowing only Google-Extended
leaves Search indexing intact while opting out of generative-model training use. Confusing the
two is exactly the kind of search-versus-training mismatch CrawlPact’s conflict detector looks for.
Example robots.txt configuration
To disallow Google-Extended specifically, without affecting any other crawler:
User-agent: Google-Extended
Disallow: /If no dedicated User-agent: Google-Extended group exists in a domain'srobots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works. A robots.txt rule is a published instruction; it does not technically prevent access.
Official source
https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
Verified against the source above as of 30 July 2026 — see how CrawlPact verifies crawler information.
Related guides
- Blocking AI training while staying visible in AI searchChoosing between CrawlPact's presets when the goal is opting out of model training without losing AI-search discoverability.
- Google-Extended vs. Googlebot: what each one actually controlsA decision guide for separating AI-training opt-out from Search indexing when configuring robots.txt for Google's crawlers.
- How to block only AI training crawlers, without blocking AI searchA step-by-step implementation guide for writing robots.txt groups that restrict training-purpose crawlers while leaving search-purpose crawlers untouched.
Other Google crawlers
- Google-CloudVertexBotCrawls requested by site owners for building Vertex AI Agents.
- GooglebotGoogle's primary web crawler for Search indexing — not an AI-training-specific crawler.
- GoogleOtherA generic Google crawler various internal product teams may use — Google's own documentation does not specify which teams or purposes.
Other training crawlers
See how this applies to your own site
Run a free audit to check your declared AI crawler policy against your own domain, or use the AI crawler checker to see how your robots.txt treats Google-Extended and every other documented crawler.