ClaudeBot vs. Claude-User vs. Claude-SearchBot: which should you block?

Like OpenAI, Anthropic documents three separate crawler tokens rather than one combined identity. Each is triggered differently and each needs its own robots.txt decision.

Which Anthropic crawler should I block?

To opt out of Anthropic’s model training while staying visible in Claude, disallow ClaudeBot and leave Claude-SearchBot and Claude-User allowed. Anthropic says blocking either of those may reduce your site’s visibility in Claude’s search results or in answers to user questions.

What does each Anthropic token do?

• ClaudeBot — training. Collects publicly accessible web content that could contribute to training Anthropic’s models.
• Claude-User — user-triggered. Fetches a page because a person directed Claude to access it as part of a query.
• Claude-SearchBot — search. Navigates the web to improve the relevance and accuracy of Claude’s search results.

What happens when you block each one?

• Disallow ClaudeBot to opt out of having your content used for Anthropic’s model training, without affecting user-triggered fetches or search relevance.
• Leave Claude-User allowed unless you want to prevent Claude from ever fetching your pages on a user’s direct request — this is a per-request, user-initiated fetch, not a bulk crawl.
• Leave Claude-SearchBot allowed if you want your pages to inform Claude’s search results.

Does a ClaudeBot rule cover Anthropic’s other crawlers?

Anthropic’s own documentation gives a single example — User-agent: ClaudeBot / Disallow: / — which some site owners assume also covers Claude-User and Claude-SearchBot because the example only names one token. It doesn’t: each token is matched independently. CrawlPact’s AI crawler checker evaluates all three against your robots.txt so any unintended gap between them is visible before it matters.