CrawlPactResources
Guides
Practical guides for AI crawler policy — decide an approach, implement it correctly, and troubleshoot it when the deployed result differs from the intent.
Decision guides
Choosing what to allow, restrict or leave unspecified — and what each choice affects.
Amazonbot vs. Amzn-SearchBot vs. Amzn-User: which should you block?
Amazon runs three crawler tokens with different purposes and different robots.txt compliance. How to configure Amazonbot, Amzn-SearchBot and Amzn-User.
Applebot vs. Applebot-Extended: Search/Siri vs. Apple Intelligence
Apple separates its long-standing search crawler from a newer, generative-AI-specific opt-out token. A decision guide for telling them apart.
Blocking AI training while staying visible in AI search
Choosing between CrawlPact's presets when the goal is opting out of model training without losing AI-search discoverability.
ClaudeBot vs. Claude-User vs. Claude-SearchBot: which should you block?
Anthropic operates three separate crawler tokens for training, user-triggered retrieval, and search. A decision guide for configuring each independently.
GPTBot vs. OAI-SearchBot vs. ChatGPT-User: which should you block?
OpenAI operates three separate crawler tokens for three separate purposes. A decision guide for choosing which, if any, to disallow.
Meta's four crawlers explained: ExternalAgent, WebIndexer, ExternalAds, ExternalFetcher
Meta documents four separate crawler tokens with four separate purposes. A decision guide for telling them apart before writing a robots.txt rule.
PerplexityBot vs. Perplexity-User: which should you block?
Perplexity separates search indexing from user-triggered fetching, and only one honours robots.txt. How to configure PerplexityBot and Perplexity-User.
robots.txt vs. meta robots tag vs. X-Robots-Tag: which mechanism to use
Three different mechanisms can restrict crawler access, at different scopes and for different content types. A decision guide for choosing between them.
RSL vs. Content Signals vs. robots.txt: which one wins when they disagree
robots.txt, RSL and Content Signals can each express AI training permissions. How to keep the three consistent, and what happens when they disagree.
Should you block CCBot? What it does and doesn't stop
CCBot builds the open Common Crawl corpus, which many other AI model trainers reuse. A decision guide for what blocking it actually achieves.
Google-Extended vs. Googlebot: what each one actually controls
A decision guide for separating AI-training opt-out from Search indexing when configuring robots.txt for Google's crawlers.
Implementation guides
Setting a decided policy up correctly in robots.txt, headers and related signals.
How to block only AI training crawlers, without blocking AI search
A step-by-step implementation guide for writing robots.txt groups that restrict training-purpose crawlers while leaving search-purpose crawlers untouched.
How to publish an llms.txt file
A step-by-step implementation guide for adding a valid llms.txt (and llms-full.txt) file to your site.
How to publish an RSL (Really Simple Licensing) declaration
Write an RSL 1.0 licence file, associate it through robots.txt, an HTTP Link header or HTML, and check it — with the official vocabulary and examples.
How to set Content Signals: the robots.txt directive and the Content-Signal header
A step-by-step guide to publishing Content Signals: the documented robots.txt Content-Signal directive, the optional use field, and the HTTP header variant.
robots.txt syntax basics for AI crawler tokens
How User-agent groups, Allow, Disallow, and Sitemap directives are evaluated, and where AI crawler tokens fit in.
Troubleshooting guides
When the deployed policy does not behave the way it was intended to.
A crawler shows 'Resource unavailable' — troubleshooting
What it means when CrawlPact can't determine a crawler's access result because robots.txt itself couldn't be fetched, and how to fix it.
My llms.txt file isn't validating — troubleshooting
Common structural issues CrawlPact's llms.txt validator flags, and how to fix each one.
My Policy Health Score dropped between scans — troubleshooting
What causes a saved domain's score to change on a scheduled re-scan, and how to find exactly what changed.
My robots.txt rule isn't blocking a crawler — troubleshooting
Common causes when a Disallow rule doesn't appear to take effect against a specific AI crawler, and how to check each one.
My RSL or Content Signals declaration isn't being detected — troubleshooting
Common reasons CrawlPact reports no RSL licence or no Content Signals for a site that publishes them, and how to check each one.