CrawlPactGuides
Practical guides for configuring and troubleshooting AI crawler policy — decision guides for choosing an approach, implementation guides for setting it up, and troubleshooting guides for when something doesn't behave as expected.
Amazonbot vs. Amzn-SearchBot vs. Amzn-User: which should you block?
Amazon documents three separate crawler tokens with different purposes and, critically, different robots.txt compliance. A decision guide for configuring each independently.
Applebot vs. Applebot-Extended: Search/Siri vs. Apple Intelligence
Apple separates its long-standing search crawler from a newer, generative-AI-specific opt-out token. A decision guide for telling them apart.
Blocking AI training while staying visible in AI search
Choosing between CrawlPact's presets when the goal is opting out of model training without losing AI-search discoverability.
ClaudeBot vs. Claude-User vs. Claude-SearchBot: which should you block?
Anthropic operates three separate crawler tokens for training, user-triggered retrieval, and search. A decision guide for configuring each independently.
A crawler shows 'Resource unavailable' — troubleshooting
What it means when CrawlPact can't determine a crawler's access result because robots.txt itself couldn't be fetched, and how to fix it.
GPTBot vs. OAI-SearchBot vs. ChatGPT-User: which should you block?
OpenAI operates three separate crawler tokens for three separate purposes. A decision guide for choosing which, if any, to disallow.
How to block only AI training crawlers, without blocking AI search
A step-by-step implementation guide for writing robots.txt groups that restrict training-purpose crawlers while leaving search-purpose crawlers untouched.
How to publish an llms.txt file
A step-by-step implementation guide for adding a valid llms.txt (and llms-full.txt) file to your site.
How to publish an RSL (Really Simple Licensing) declaration
A step-by-step implementation guide for declaring permitted and prohibited AI uses of your content via RSL.
How to set the Content-Signal HTTP header
A step-by-step implementation guide for publishing a Content-Signal response header, and where to configure it depending on your hosting setup.
My llms.txt file isn't validating — troubleshooting
Common structural issues CrawlPact's llms.txt validator flags, and how to fix each one.
Meta's four crawlers explained: ExternalAgent, WebIndexer, ExternalAds, ExternalFetcher
Meta documents four separate crawler tokens with four separate purposes. A decision guide for telling them apart before writing a robots.txt rule.
PerplexityBot vs. Perplexity-User: which should you block?
Perplexity separates its search-indexing crawler from its user-triggered fetcher — and only one of them actually honours robots.txt. A decision guide for configuring each independently.
My Policy Health Score dropped between scans — troubleshooting
What causes a saved domain's score to change on a scheduled re-scan, and how to find exactly what changed.
My robots.txt rule isn't blocking a crawler — troubleshooting
Common causes when a Disallow rule doesn't appear to take effect against a specific AI crawler, and how to check each one.
robots.txt vs. meta robots tag vs. X-Robots-Tag: which mechanism to use
Three different mechanisms can restrict crawler access, at different scopes and for different content types. A decision guide for choosing between them.
My RSL or Content Signals declaration isn't being detected — troubleshooting
Common causes when CrawlPact reports 'not found' for a published RSL declaration or Content-Signal header, and how to check each one.
RSL vs. Content Signals vs. robots.txt: which one wins when they disagree
Three separate, overlapping declarations can express AI training permissions. A decision guide for keeping them consistent and understanding what happens when they don't agree.
Should you block CCBot? What it does and doesn't stop
CCBot builds the open Common Crawl corpus, which many other AI model trainers reuse. A decision guide for what blocking it actually achieves.
Google-Extended vs. Googlebot: what each one actually controls
A decision guide for separating AI-training opt-out from Search indexing when configuring robots.txt for Google's crawlers.
robots.txt syntax basics for AI crawler tokens
How User-agent groups, Allow, Disallow, and Sitemap directives are evaluated, and where AI crawler tokens fit in.