CrawlPactAnthropic · Training crawler
ClaudeBot
Used by Anthropic to crawl publicly accessible content for model training.
Questions about ClaudeBot
What is ClaudeBot?
ClaudeBot is Anthropic's crawler for web content that could contribute to model training. Anthropic documents two other bots with their own tokens: Claude-SearchBot (search quality) and Claude-User (pages a user's question needs).
What does disallowing ClaudeBot do?
Anthropic says it signals that your site's future materials should be excluded from its AI model training datasets. Claude-SearchBot and Claude-User are addressed by their own robots.txt groups.
Does ClaudeBot support Crawl-delay?
Anthropic says it supports the non-standard Crawl-delay extension to robots.txt.
ClaudeBot is documented by Anthropic as the crawler used to gather publicly available web
content for training its Claude models.
What blocking ClaudeBot does
Disallowing ClaudeBot in robots.txt signals that a website does not want its content used for
model training. It does not affect Anthropic’s other documented crawlers, Claude-User
(user-triggered retrieval) or Claude-SearchBot (search relevance), which are evaluated and
reported separately by CrawlPact — a website can restrict training while remaining discoverable
through Claude’s search or user-directed features.
Site-owner controls
Anthropic documents standard robots.txt support for disallowing ClaudeBot. As with other
training-purpose crawlers, blocking it is a declared-policy signal only — see
/limitations for what a robots.txt rule can and cannot guarantee.
How to verify ClaudeBot requests
Anthropic publishes its crawlers' IP addresses at https://claude.com/crawling/bots.json, and recommends robots.txt rather than IP blocking as the way to opt out. Operator documentation.
A user-agent string alone can be copied by anyone; confirming a request really came from Anthropic needs your server or CDN logs. CrawlPact audits declared policy and does not see your traffic.
Example robots.txt configuration
To disallow ClaudeBot specifically, without affecting any other crawler:
User-agent: ClaudeBot
Disallow: /If no dedicated User-agent: ClaudeBot group exists in a domain'srobots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works. A robots.txt rule is a published instruction; it does not technically prevent access.
Official source
Verified against the source above as of 24 July 2026 — see how CrawlPact verifies crawler information.
Related guides
- Blocking AI training while staying visible in AI searchChoosing between CrawlPact's presets when the goal is opting out of model training without losing AI-search discoverability.
- ClaudeBot vs. Claude-User vs. Claude-SearchBot: which should you block?Anthropic operates three separate crawler tokens for training, user-triggered retrieval, and search. A decision guide for configuring each independently.
- How to block only AI training crawlers, without blocking AI searchA step-by-step implementation guide for writing robots.txt groups that restrict training-purpose crawlers while leaving search-purpose crawlers untouched.
Other Anthropic crawlers
Other training crawlers
See how this applies to your own site
Run a free audit to check your declared AI crawler policy against your own domain, or use the AI crawler checker to see how your robots.txt treats ClaudeBot and every other documented crawler.