Google-CloudVertexBot

Google’s own crawler documentation describes Google-CloudVertexBot as affecting “crawls requested by site owners for building Vertex AI Agents.” Unlike Googlebot or Google-Extended, this crawler acts on a specific site owner’s own request, rather than a general web-wide sweep.

Why this is categorised as “agent”

Google’s documentation frames this crawler around a site owner’s own request to build an AI agent against their content, which fits CrawlPact’s registry’s “agent” purpose category more than a general search or bulk-training crawl.

Site-owner controls

Disallowing Google-CloudVertexBot prevents crawls performed specifically because a site owner (not necessarily you) requested that a Vertex AI Agent be built against this content. It does not affect Google Search indexing (Googlebot) or the generative-AI-training opt-out (Google-Extended) — this token governs only agent-initiated, request-driven access, not a general web-wide crawl. See /limitations for what a robots.txt rule can and cannot guarantee.

Example robots.txt configuration

To disallow Google-CloudVertexBot specifically, without affecting any other crawler:

User-agent: Google-CloudVertexBot
Disallow: /

If no dedicated User-agent: Google-CloudVertexBot group exists in a domain'srobots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works. A robots.txt rule is a published instruction; it does not technically prevent access.

Official source

https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

Verified against the source above as of 24 July 2026 — see how CrawlPact verifies crawler information.