CrawlPactRobots.txt AI validator
Enter a domain to validate its live robots.txt the way RFC 9309 defines it — groups, rules, sitemap lines and syntax problems, each with its line number — and then see what those rules mean for documented AI crawlers.
Runs the same real audit as the full report and opens it at this check — free, with no account or installation.
What this checks
- Parses every line: User-agent groups, Allow and Disallow rules, Sitemap lines, comments and fields outside RFC 9309.
- Reports syntax problems by line: a rule before any User-agent line, a line without a field name, an invalid Sitemap URL, an HTML page served instead of robots.txt, invalid encoding and oversized files.
- Notes a token that appears in more than one group (RFC 9309 combines them) and a wildcard group that blocks everything.
- Evaluates every registry AI crawler against the parsed rules and proposes copyable additions for the objective you choose.
Worked example
Disallow: /tmp/
User-agent: *
Allow: /
Sitemap: /sitemap.xml
Content-Signal: ai-train=noWhat the report shows
| Line 1 | WarningA rule before any User-agent group is ignored |
|---|---|
| Line 4 | WarningSitemap must be an absolute URL |
| Line 5 | NoteContent-Signal is an extension, evaluated as Content Signals |
| Groups | 1User-agent: * — lines 2–3 |
How to read the result
- Error
- The file cannot be evaluated as robots.txt (for example, an HTML page was returned).
- Warning
- A line is ignored or likely to be read differently than intended.
- Note
- Context, not a defect — for example an extension field such as Content-Signal or License, which is evaluated in its own report section.
What this does not tell you
- Matching follows RFC 9309 and common crawler behaviour; an individual crawler can interpret edge cases differently.
- Only the live robots.txt at the domain's root is read. Subdomains each have their own file.
- Valid syntax says nothing about whether crawlers comply.
See limitations andmethodology.
Questions
- Does a valid robots.txt mean AI crawlers will follow it?
- No. RFC 9309 defines the format; compliance is up to each crawler operator. The validator tells you what the file says and whether it is well-formed.
- Are Content-Signal and License lines errors?
- No. They are extensions outside RFC 9309 and do not change Allow or Disallow evaluation. CrawlPact reads them separately as Content Signals and RSL declarations.
- What happens if my site has no robots.txt?
- RFC 9309 treats an unavailable (4xx) robots.txt as allowing everything. CrawlPact reports the file as unavailable rather than inventing rules.
Related
Sources
Want the complete picture — and to know when it changes?
This check is one section of the full audit, which also covers every other policy signal and a Policy Health Score. Save the domain from the report to monitor it.