CrawlPactCrawlPact audits declared policy
CrawlPact reports what a website's public configuration asks crawlers to do (declared policy) and what CrawlPact itself was able to retrieve (observed response). It does not, and cannot, prove what an external crawler actually does (actual crawler behaviour) —robots.txt and related signals are voluntary, unenforced declarations.
Some crawlers document exceptions to robots.txt itself
Beyond the general point above, a small number of governed crawlers — currentlyChatGPT-User,Meta-ExternalFetcher,Amzn-User, andPerplexity-User— have their own vendor documentation stating that a robots.txt rule may not reliably prevent their access, typically because the request is a single fetch made on a person's behalf rather than a bulk crawl. CrawlPact surfaces this as a caveat directly on the audit result row and on that crawler's own page rather than only here; see each crawler's page for the exact vendor wording.
CrawlPact is not
- A web application firewall or reverse proxy
- A live crawler blocker
- A server-log analytics service
- A full-site SEO crawler or general-purpose SEO suite
- An AI-search ranking tracker or brand-mention tracker
- A legal service or compliance certificate
- A guarantee that bots will obey instructions or that content will appear in AI answers
Several of these jobs are done well by other products. How CrawlPact sits next to edge enforcement, bot analytics, AI-visibility and licensing tools is set out in Compare.
Evidence can be incomplete
A target may block CrawlPact's scanner, return an unexpected response, or expose only partial signals. When evidence is insufficient, CrawlPact shows "Incomplete" rather than a misleading score or result.
Registry coverage is not exhaustive
CrawlPact's crawler registry only includes crawlers with a reliable official source and a verified token. An unlisted or newly introduced crawler will not appear in a report until it is verified and published.