CrawlPactFor Publishers
AI crawler policy for publishers
Search and AI-training crawlers often warrant different decisions, but a site's public signals frequently don't distinguish them clearly — and CDN or deployment changes can alter the actual response without an editorial decision being made.
The audit is free — no account and nothing to install.
Pro plan
$19 / monthor $189 / year
- 25 saved domains
- Weekly automatic monitoring
- 24 months of audit history
The problem
Where it usually goes wrong
One rule for “AI bots generally”
A robots.txt rule written against a single token, while other documented tokens for the same or a different purpose remain unaddressed.
Signals that contradict each other
A meta robots tag or X-Robots-Tag header that says something different from robots.txt for the same crawler.
A platform quietly rewrites the file
A CDN, caching layer or CMS plugin regenerating or overriding a previously set robots.txt after a migration or theme change.
Search access lost by accident
An overly broad rule meant for training crawlers that also restricts the search crawlers you rely on for visibility.
The workflow
How CrawlPact fits the work
Audit, keep the evidence, monitor for change. Every decision about what to change stays with you.
Step 1: Fetch the live response
robots.txt, relevant meta directives and header signals — exactly as your site serves them today.
Step 2: Classify every crawler by purpose
A versioned, source-cited registry separates search, training, user-triggered retrieval, agent, advertising-validation and research crawlers — and records “unspecified” honestly when an operator hasn't documented one.
Step 3: Report conflicts with evidence
Every finding cites what was fetched, so editorial and technical teams can verify it independently rather than taking CrawlPact's word for it.
Step 4: Decide, then change it yourself
Recommendations describe what a rule change would do. Applying it is always a manual, deliberate step.
Step 5: Monitor for drift
Rechecks flag a change to your site's published policy and a change to the crawler registry itself.
In depth
Why publishers face a distinct decision
A publisher’s core tension is that “AI crawler” isn’t one category. Search crawlers, AI-training crawlers, retrieval/agent crawlers acting on a reader’s behalf, and advertising-validation crawlers can all request the same URL — and a publisher may reasonably want to stay visible in AI-powered search while declining to have the same content used for model training, or vice versa. Getting that distinction right requires knowing, specifically, which documented crawler tokens fall into which category, and confirming that the site’s actual published rules reflect the intended decision — not assuming they do.
Why registry changes matter to publishers
A recheck flags both a change to the site’s own published policy and a change to CrawlPact’s verified crawler registry — relevant for publishers specifically because a registry correction (a newly documented crawler, or a corrected purpose classification) can change what a previously-reviewed policy effectively means, with zero change on the publisher’s own site.
Sharing findings inside the newsroom
Private, revocable report sharing is available on every plan, including Free — useful for sharing a finding between editorial and technical teams without publishing it. Manual re-scans let you recheck immediately after a deliberate change rather than waiting for the next scheduled cycle.
The proof boundary
What CrawlPact does — and does not do
CrawlPact audits and monitors the public policy signals a website publishes. It does not control crawlers or guarantee that any of them comply.
What it does
- Distinguishes search, training, retrieval and agent crawlers by their documented purpose
- Attaches the fetched evidence to every finding
- Keeps a dated history of how your published policy changed
What it does not do
- Block scraping, monetise content or license content on your behalf
- Give legal advice, or a single recommended policy that fits every publisher
- Guarantee that any crawler complies — robots.txt and related signals are a publicly stated request, not enforcement
The right balance between AI-search visibility and training-data exclusion is an editorial decision. CrawlPact informs it with evidence; it does not make it.
Read the methodology for exactly what is evaluated, how the Policy Health Score is calculated, and the limitations before relying on a result.
Plan fit
Pro is the usual fit
Weekly monitoring catches a change quickly, and long audit history keeps your policy's evolution as evidence — not just its current state.
Also consider Solo. Fits a single publication that only needs a monthly recheck.
Every plan, including Free, gets the same complete, accurate audit. Paid plans add saved domains, monitoring, history, and export and sharing capacity — not a more accurate result.
| Feature | Solo | Pro |
|---|---|---|
| Monthly price | $9 | $19 |
| Yearly price | $89 | $189 |
| Saved domains | 5 | 25 |
| Automatic monitoring | Monthly | Weekly |
| Manual rescans per domain / month | 5 | 10 |
| Audit history | 12 months | 24 months |
| Private Atom change feed | Included | Included |
| Domain groups | Not included | Included |
| CSV domain export | Not included | Included |
| Batch domain import | Not included | Up to 10 |
| Agency-branded shared reports | Not included | Not included |
Questions
Frequently asked questions
Can CrawlPact guarantee my content won't be used for AI training?
No. CrawlPact reports on the public signals a site publishes requesting that outcome; it cannot guarantee any crawler's actual compliance.
Does CrawlPact distinguish search crawlers from training crawlers automatically?
Yes — the crawler registry classifies each documented token by purpose, and audit findings are organised around that distinction rather than treating “AI crawler” as one undifferentiated category.
What plan fits a single publication?
Pro typically fits a single publisher's domain. The plan section above shows its current monitoring cadence, history and price, read live from the plan catalog.
Relevant platform guides
- AI crawler policy on CloudflareHow Cloudflare's managed robots.txt and AI Crawl Control interact with your site's public AI crawler policy, verified against Cloudflare documentation.
- AI crawler policy on WordPressHow WordPress's virtual robots.txt, the 'Discourage search engines' setting and plugins change what your site tells AI crawlers, per WordPress docs.
Related guides
- Blocking AI training while staying visible in AI searchChoosing between CrawlPact's presets when the goal is opting out of model training without losing AI-search discoverability.
- How to block only AI training crawlers, without blocking AI searchA step-by-step implementation guide for writing robots.txt groups that restrict training-purpose crawlers while leaving search-purpose crawlers untouched.
- Meta's four crawlers explained: ExternalAgent, WebIndexer, ExternalAds, ExternalFetcherMeta documents four separate crawler tokens with four separate purposes. A decision guide for telling them apart before writing a robots.txt rule.
Other teams CrawlPact serves
- AgenciesAudit, explain, and monitor AI crawler policy across every client website from one place — evidence-backed, without exposing internal tooling to clients.
- SaaS and documentation teamsAudit and monitor AI crawler access to product docs, knowledge bases and marketing domains — including what deployment platforms change for you.
- Web developersConfirm the AI crawler policy you deployed matches what you intended, across frameworks, hosting platforms and CDNs that rewrite public files.
See what your own site tells AI crawlers
Run a free audit on any public domain. The report shows every finding with the evidence behind it — the same complete audit on every plan.