CrawlPactAnthropic · Search crawler
Claude-SearchBot
Navigates the web to improve the relevance and accuracy of Claude's search results.
Questions about Claude-SearchBot
What happens if I block Claude-SearchBot?
Anthropic says your content then won't be indexed for search, which may reduce your site's visibility and accuracy in Claude's search results. It does not opt your content out of training — that is ClaudeBot.
Anthropic documents Claude-SearchBot as a crawler that “navigates the web to improve search
result quality for users,” analyzing content specifically to enhance the relevance and accuracy
of search responses — distinct from ClaudeBot’s training purpose.
Why the distinction matters
A website that wants to remain discoverable through Claude’s search features while restricting
training use of its content needs to treat Claude-SearchBot and ClaudeBot as separate
decisions in robots.txt.
Site-owner controls
Anthropic’s own documentation gives this example for blocking any of its crawlers:
User-agent: Claude-SearchBot
Disallow: /See /limitations for what a robots.txt rule can and cannot guarantee.
How to verify Claude-SearchBot requests
Anthropic publishes its crawlers' IP addresses at https://claude.com/crawling/bots.json. Operator documentation.
A user-agent string alone can be copied by anyone; confirming a request really came from Anthropic needs your server or CDN logs. CrawlPact audits declared policy and does not see your traffic.
Example robots.txt configuration
To disallow Claude-SearchBot specifically, without affecting any other crawler:
User-agent: Claude-SearchBot
Disallow: /If no dedicated User-agent: Claude-SearchBot group exists in a domain'srobots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works. A robots.txt rule is a published instruction; it does not technically prevent access.
Official source
Verified against the source above as of 24 July 2026 — see how CrawlPact verifies crawler information.
Related guides
- Blocking AI training while staying visible in AI searchChoosing between CrawlPact's presets when the goal is opting out of model training without losing AI-search discoverability.
- ClaudeBot vs. Claude-User vs. Claude-SearchBot: which should you block?Anthropic operates three separate crawler tokens for training, user-triggered retrieval, and search. A decision guide for configuring each independently.
Other Anthropic crawlers
Other search crawlers
See how this applies to your own site
Run a free audit to check your declared AI crawler policy against your own domain, or use the AI crawler checker to see how your robots.txt treats Claude-SearchBot and every other documented crawler.