CrawlPact

AI crawler checker

Enter a domain to see whether documented AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and others in CrawlPact's registry — appear allowed or blocked, with the exact matched robots.txt rule and line for each.

What this checks

  • Runs a full CrawlPact scan of the domain and takes you straight to the crawler-by-crawler findings.
  • For each crawler in CrawlPact's registry, shows whether it appears allowed or blocked by the retrieved robots.txt, and the exact matched rule and line.
  • Distinguishes search, training, user-triggered, and agent crawlers from the same operator (for example OpenAI's OAI-SearchBot, GPTBot, and ChatGPT-User), since blocking one does not affect the others.

What this does not check

This reports declared policy and CrawlPact's own retrieved response — not whether any crawler actually obeys it, and not crawlers absent from CrawlPact's registry. See limitations for the full list of boundaries.

Related

Want the complete picture?

This checker is a view into a full CrawlPact audit, which also covers llms.txt, RSL, Content Signals, and your Policy Health Score.

Run the full audit

Analytics preferences

CrawlPact uses optional Google Analytics and Microsoft Clarity on public marketing pages to understand which content is useful and how visitors actually use it. Neither is used in the authenticated app or admin areas. You can accept or decline analytics without affecting the service.