Anthropic

ClaudeBot

User-agent token: ClaudeBot
Purpose: Training
Status: active
Last verified: 2026-07-24

ClaudeBot is documented by Anthropic as the crawler used to gather publicly available web content for training its Claude models.

Site-owner controls

Anthropic documents standard robots.txt support for disallowing ClaudeBot. As with other training-purpose crawlers, blocking it is a declared-policy signal only — see /limitations for what a robots.txt rule can and cannot guarantee.

Example robots.txt configuration

To disallow ClaudeBot specifically, without affecting any other crawler:

User-agent: ClaudeBot
Disallow: /

If no dedicated User-agent: ClaudeBot group exists in a domain's robots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works.

Official source: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

Verified against the source above as of 2026-07-24 — see how CrawlPact verifies crawler information.

Related guides

ClaudeBot vs. Claude-User vs. Claude-SearchBot: which should you block?
Anthropic operates three separate crawler tokens for training, user-triggered retrieval, and search. A decision guide for configuring each independently.

Related crawlers

See how this applies to your own site

Run a free audit to check your declared AI crawler policy against your own domain, or use the AI crawler checker to check this one crawler specifically.

Audit a domain

Analytics preferences

CrawlPact uses optional Google Analytics and Microsoft Clarity on public marketing pages to understand which content is useful and how visitors actually use it. Neither is used in the authenticated app or admin areas. You can accept or decline analytics without affecting the service.