Googlebot

Questions about Googlebot

What is Googlebot?

Googlebot is Google's crawler for building its search indexes. Google says its robots.txt rules affect Google Search and other products such as Images, Video, News and Discover.

Does blocking Googlebot keep my pages out of AI Overviews?

Google's AI features are part of Google Search, and Google names robots.txt rules for Googlebot as the crawl control for them. Blocking Googlebot therefore affects ordinary search results too; Google's snippet controls (nosnippet, max-snippet) limit what is shown instead.

Google’s Search Central documentation describes Googlebot as “the generic name for two types of web crawlers used by Google Search”: a desktop crawler and a mobile crawler that simulate a user browsing a site, used to crawl and index content for Google Search results.

Why CrawlPact tracks Googlebot

Googlebot is not an AI-training crawler — it exists for conventional search indexing. CrawlPact includes it in the registry because a website’s overall crawler posture (which bots are allowed, and why) is easier to reason about when a general-purpose search crawler is shown alongside AI-specific ones like Google-Extended, which separately controls use of content for training Gemini and Vertex AI generative models.

A note on this record

Google’s own page describes how to identify Googlebot subtypes by their user-agent header but does not spell out a single literal token string on that page — Googlebot is the name Google uses consistently across its own documentation and is the value CrawlPact matches against in robots.txt.

Site-owner controls

Disallowing Googlebot removes a page from Google Search indexing entirely — the most consequential single crawler decision on this list for most public websites, since it governs organic search visibility rather than any AI-training-specific use. It is separate from Google-Extended (generative-AI training opt-out) and Google-CloudVertexBot (site-owner-requested Vertex AI Agent crawls): blocking either of those does not affect Search indexing, and blocking Googlebot does not, by itself, opt content out of the AI training use Google-Extended governs. See /limitations for what a robots.txt rule can and cannot guarantee.

How to verify Googlebot requests

Google says the user agent can be spoofed. Its common crawlers use IP ranges published in common-crawlers.json, and their reverse DNS hostnames end in googlebot.com (crawl-…googlebot.com or geo-crawl-…geo.googlebot.com). Operator documentation.

A user-agent string alone can be copied by anyone; confirming a request really came from Google needs your server or CDN logs. CrawlPact audits declared policy and does not see your traffic.

Example robots.txt configuration

To disallow Googlebot specifically, without affecting any other crawler:

User-agent: Googlebot
Disallow: /

If no dedicated User-agent: Googlebot group exists in a domain'srobots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works. A robots.txt rule is a published instruction; it does not technically prevent access.

Official source

https://developers.google.com/search/docs/crawling-indexing/googlebot

Verified against the source above as of 24 July 2026 — see how CrawlPact verifies crawler information.