CrawlPactFor Web developers
Verify AI crawler policy after every deployment
The crawler policy a developer intends to ship and the crawler policy a framework, hosting platform, or CDN actually serves can diverge — and that gap is invisible until someone checks the live, deployed response.
The audit is free — no account and nothing to install.
Solo plan
$9 / monthor $89 / year
- 5 saved domains
- Monthly automatic monitoring
- 12 months of audit history
The problem
Where it usually goes wrong
Build output isn't your source
A framework's robots.txt helper produces different output than you expected for a given route or environment.
Preview copies production
A preview or staging deployment serves the same crawler policy as production — too permissive or too restrictive for that environment.
A stale cache after deploy
A CDN keeps serving the previous robots.txt until the cache expires or is purged.
Layers override each other
A meta robots tag or header set in application code is overridden or duplicated by a CMS plugin or CDN rule.
A token that doesn't match
A rule meant for one crawler token inadvertently matches — or misses — a related token from the same operator.
The workflow
How CrawlPact fits the work
Audit, keep the evidence, monitor for change. Every decision about what to change stays with you.
Step 1: Deploy as usual
Ship through your framework, hosting platform and CDN. There is nothing to install.
Step 2: Audit the live response
CrawlPact fetches robots.txt, meta directives and header signals the way a real crawler encounters them.
Step 3: Reproduce any finding
Every finding includes what was fetched, so you can confirm it with curl, a browser or any other tool.
Step 4: Re-scan after a deploy
A manual re-scan checks immediately, instead of waiting for the next scheduled monitoring cycle.
Step 5: Let monitoring catch the rest
Scheduled rechecks flag a change caused by a platform update or CDN rule with no corresponding commit.
In depth
Why the deployed response is the only thing that matters
A robots.txt file committed to a repository, a meta tag written in a template, or a header set
in application code is an intent. What a crawler actually receives is the deployed response —
and frameworks, static-site generators, hosting platforms, and CDNs can all sit between the two.
A build step can generate a different robots.txt than the one in source control. A platform
default can apply when no file is explicitly served. A CDN edge rule or cache can serve a stale or
modified version of what origin actually returns. None of this is unusual or a bug in any single
tool — it’s just several independent layers each capable of affecting the same output, which makes
“check what I wrote” a different question from “check what’s actually live.”
Signals CrawlPact checks
CrawlPact fetches the actual public response for a domain — robots.txt, relevant meta
directives, and header signals — and evaluates it against a versioned, source-cited crawler
registry. See the crawler directory for exact documented tokens,
Scanner information for the technical detail behind the fetch itself, and the
platform guides for what’s independently verifiable about how specific hosting
platforms and CDNs handle these signals.
The proof boundary
What CrawlPact does — and does not do
CrawlPact audits and monitors the public policy signals a website publishes. It does not control crawlers or guarantee that any of them comply.
What it does
- Fetches the deployed response — not your source files
- Flags conflicts between robots.txt, meta and header layers as explicit findings
- Attaches the fetched evidence so you can reproduce every finding
What it does not do
- Deploy, edit or roll back your robots.txt, meta tags or headers on any platform
- Run on every deploy or monitor in real time — a manual re-scan is the way to check a specific deploy
- Enforce access control or block traffic like a WAF
- Replace testing your robots.txt before it ships — it catches what changes after deployment
Read the methodology for exactly what is evaluated, how the Policy Health Score is calculated, and the limitations before relying on a result.
Plan fit
Solo is the usual fit
Sized for one developer or project: monthly automatic monitoring, a private Atom feed for change alerts, and manual re-scans for checking right after a deploy.
Also consider Pro. Fits several projects or client sites that need weekly rechecks.
Every plan, including Free, gets the same complete, accurate audit. Paid plans add saved domains, monitoring, history, and export and sharing capacity — not a more accurate result.
| Feature | Solo | Pro |
|---|---|---|
| Monthly price | $9 | $19 |
| Yearly price | $89 | $189 |
| Saved domains | 5 | 25 |
| Automatic monitoring | Monthly | Weekly |
| Manual rescans per domain / month | 5 | 10 |
| Audit history | 12 months | 24 months |
| Private Atom change feed | Included | Included |
| Domain groups | Not included | Included |
| CSV domain export | Not included | Included |
| Batch domain import | Not included | Up to 10 |
| Agency-branded shared reports | Not included | Not included |
Questions
Frequently asked questions
Does CrawlPact edit my production configuration for me?
No. CrawlPact is not a deployment platform and does not modify your site's configuration — it audits the public response and reports findings; applying a fix is a step you take in your own codebase or platform.
Does this replace testing my robots.txt before deploy?
No — CrawlPact checks the live, deployed response, which complements (not replaces) checking a file locally before it ships. The value is specifically in catching what changes after deployment, at the platform or CDN layer, which local testing can't see.
Does CrawlPact provide WAF-style enforcement or blocking?
No. CrawlPact audits and reports on publicly declared policy signals; it does not enforce access control at the network or application layer.
Relevant platform guides
- AI crawler policy on CloudflareHow Cloudflare's managed robots.txt and AI Crawl Control interact with your site's public AI crawler policy, verified against Cloudflare documentation.
- AI crawler policy on NetlifyHow Netlify's deploy-preview noindex header, _headers file and deploy contexts change what your site tells AI crawlers, per Netlify's documentation.
- AI crawler policy on ShopifyHow Shopify's robots.txt.liquid customization mechanism works, its default rules, and its documented limits — verified against official Shopify documentation.
- AI crawler policy on VercelHow Vercel's preview noindex header, static or generated robots.txt and custom headers change what your site tells AI crawlers, per official docs.
- AI crawler policy on WordPressHow WordPress's virtual robots.txt, the 'Discourage search engines' setting and plugins change what your site tells AI crawlers, per WordPress docs.
Related guides
- A crawler shows 'Resource unavailable' — troubleshootingWhat it means when CrawlPact can't determine a crawler's access result because robots.txt itself couldn't be fetched, and how to fix it.
- My Policy Health Score dropped between scans — troubleshootingWhat causes a saved domain's score to change on a scheduled re-scan, and how to find exactly what changed.
- My robots.txt rule isn't blocking a crawler — troubleshootingCommon causes when a Disallow rule doesn't appear to take effect against a specific AI crawler, and how to check each one.
Other teams CrawlPact serves
- AgenciesAudit, explain, and monitor AI crawler policy across every client website from one place — evidence-backed, without exposing internal tooling to clients.
- PublishersAudit and monitor the difference between what your site tells AI search crawlers and what it tells AI training crawlers — with evidence, not guesswork.
- SaaS and documentation teamsAudit and monitor AI crawler access to product docs, knowledge bases and marketing domains — including what deployment platforms change for you.
See what your own site tells AI crawlers
Run a free audit on any public domain. The report shows every finding with the evidence behind it — the same complete audit on every plan.