Google

Google-Extended

User-agent token: Google-Extended
Purpose: Training
Status: active
Last verified: 2026-07-30

Google-Extended is a control token, documented by Google, that lets a website opt out of having its content used to train Gemini and Vertex AI generative APIs — without affecting Google Search indexing, which is governed separately by Googlebot.

Why this one is easy to get wrong

Disallowing Googlebot blocks Search indexing entirely. Disallowing only Google-Extended leaves Search indexing intact while opting out of generative-model training use. Confusing the two is one of the most common policy conflicts CrawlPact’s conflict detector looks for.

Example robots.txt configuration

To disallow Google-Extended specifically, without affecting any other crawler:

User-agent: Google-Extended
Disallow: /

If no dedicated User-agent: Google-Extended group exists in a domain's robots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works.

Official source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

Verified against the source above as of 2026-07-30 — see how CrawlPact verifies crawler information.

Related guides

Google-Extended vs. Googlebot: what each one actually controls
A decision guide for separating AI-training opt-out from Search indexing when configuring robots.txt for Google's crawlers.

Related crawlers

See how this applies to your own site

Run a free audit to check your declared AI crawler policy against your own domain, or use the AI crawler checker to check this one crawler specifically.

Audit a domain

Analytics preferences

CrawlPact uses optional Google Analytics and Microsoft Clarity on public marketing pages to understand which content is useful and how visitors actually use it. Neither is used in the authenticated app or admin areas. You can accept or decline analytics without affecting the service.