CrawlPactImplementation guide
robots.txt syntax basics for AI crawler tokens
How User-agent groups, Allow, Disallow, and Sitemap directives are evaluated, and where AI crawler tokens fit in.
robots.txt is a plain-text file at the root of a domain (/robots.txt) made up of one or more
User-agent groups, each containing Allow and Disallow directives, plus optional Sitemap
declarations.
Group matching
A crawler matches the most specific User-agent group that names its token exactly; if no exact
match exists, it falls back to the wildcard (User-agent: *) group. Within a matching group, the
longest matching path rule wins when Allow and Disallow rules conflict for the same path —
this is the RFC 9309 behaviour CrawlPact’s parser follows (see /methodology).
Adding an AI-specific rule
User-agent: GPTBot
Disallow: /This creates a dedicated group for a single crawler token without affecting the wildcard group that other, unlisted crawlers fall back to. CrawlPact’s recommendation engine generates rules in this shape, preserving unrelated existing directives.
Related guides
- How to block only AI training crawlers, without blocking AI searchA step-by-step implementation guide for writing robots.txt groups that restrict training-purpose crawlers while leaving search-purpose crawlers untouched.
- How to publish an llms.txt fileA step-by-step implementation guide for adding a valid llms.txt (and llms-full.txt) file to your site.
- How to publish an RSL (Really Simple Licensing) declarationWrite an RSL 1.0 licence file, associate it through robots.txt, an HTTP Link header or HTML, and check it — with the official vocabulary and examples.
See how this applies to your own site
Run a free audit to check your declared AI crawler policy against your own domain.