CrawlPactAI crawler checker
Enter a domain to see how its live robots.txt addresses every crawler in CrawlPact's source-verified registry — grouped by purpose, so a search crawler and a training crawler from the same company are never treated as one.
Runs the same real audit as the full report and opens it at this check — free, with no account or installation.
What this checks
- Retrieves your robots.txt and evaluates each registry crawler's user-agent token against it, following RFC 9309 group selection.
- Groups results by purpose: search, training, user-triggered retrieval, agents, and other documented uses.
- Shows the matched rule and its line number for every allowed or blocked result.
- Checks the results against the objective you choose — for example, a training crawler left allowed under "allow search, block training".
Worked example
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /What the report shows
| GPTBot (OpenAI, training) | BlockedDisallow: / — line 2 |
|---|---|
| OAI-SearchBot (OpenAI, search) | Allowedwildcard group, Allow: / — line 5 |
| ChatGPT-User (OpenAI, user-triggered) | Allowedwildcard group, Allow: / — line 5 |
| ClaudeBot (Anthropic, training) | Allowedwildcard group — no group names it |
How to read the result
- Allowed
- The most specific matching rule for the site root (/) allows it, or no rule matches.
- Blocked
- The most specific matching rule disallows the site root.
- No explicit rule
- No group names the crawler and there is no wildcard (*) group. That is not the same as blocked.
- Resource unavailable
- robots.txt could not be retrieved, so nothing was evaluated. Crawlers treat a missing file differently from an error — see the troubleshooting guide.
What this does not tell you
- Results are evaluated for the site root (/). A rule that only covers a sub-path (say, Disallow: /private/) does not change the root result.
- A robots.txt rule is a published instruction. CrawlPact cannot see whether a crawler visits or obeys it — that needs your server or CDN logs.
- Only crawlers in CrawlPact's registry are evaluated; each links to its operator's own documentation from the crawler directory.
See limitations andmethodology.
Questions
- Does blocking GPTBot remove my site from ChatGPT search?
- No. OpenAI documents separate tokens: GPTBot for training, OAI-SearchBot for search and ChatGPT-User for requests a user makes. The checker shows each one separately so you can see which you affect.
- Why does a crawler I never mentioned show as allowed?
- Under RFC 9309 a crawler with no group of its own follows the wildcard (User-agent: *) group. If that group allows the root, so does the result.
- Is this the same as checking my server logs?
- No. It reads what your robots.txt declares. Whether a crawler actually requested a page, or ignored the rule, only appears in server or CDN logs.
- Which AI crawlers are included?
- Every evaluation-eligible crawler in the active registry release — the crawler directory lists them with their operator, purpose and last verification date.
Related
Sources
Want the complete picture — and to know when it changes?
This check is one section of the full audit, which also covers every other policy signal and a Policy Health Score. Save the domain from the report to monitor it.