CrawlPactFor SaaS and documentation teams
AI crawler policy for SaaS and documentation teams
Documentation discoverability decisions, the search-versus-training distinction, and deployment-platform-driven policy drift all intersect on the domains SaaS teams run — often across multiple subdomains with different owners.
The audit is free — no account and nothing to install.
Pro plan
$19 / monthor $189 / year
- 25 saved domains
- Weekly automatic monitoring
- 24 months of audit history
The problem
Where it usually goes wrong
Docs live on their own deployment
docs.example.com, a dedicated Vercel or Netlify project, or a static-site generator — each with its own robots.txt, or none at all and a platform default.
Subdomains drift apart
The documentation policy drifts from what the main site intends, simply because nobody audits the two as a pair.
Platforms change what's served
Deploy previews, platform defaults and caching layers can each change the served response without a commit to your repository.
Training is a separate decision
Docs are exactly what AI search and retrieval crawlers request most. Whether they are also available for model training deserves its own deliberate choice.
The workflow
How CrawlPact fits the work
Audit, keep the evidence, monitor for change. Every decision about what to change stays with you.
Step 1: Audit each domain live
What the deployed response actually says — not what a config file should produce — with the fetched evidence attached.
Step 2: Group docs with their sites
Domain groups keep documentation, product and marketing domains together as one coherent set.
Step 3: Monitor for platform-side change
Scheduled rechecks flag a change caused by a redeploy or a platform update, not only your own commits.
Step 4: Subscribe to changes
A private Atom feed delivers change alerts to whatever feed reader or automation already consumes Atom.
Step 5: Loop in the owning team
Share a private, revocable report with whichever team — docs, platform or security — owns the affected subdomain.
In depth
Why this matters for SaaS and documentation
Product documentation and developer-facing content is exactly the kind of material AI search and retrieval crawlers request most — a reader (or an AI agent acting for one) looking up how an API works is a core discoverability case teams generally want to support. At the same time, some teams want to separately consider whether the same content should be available for AI model training. That’s a real decision to make deliberately, not a default that happens to fall out of whatever a docs platform ships by default.
Documentation discoverability risk
A documentation site is frequently a separate subdomain or deployment from the main marketing
site — docs.example.com, a dedicated Vercel/Netlify project, or a self-hosted static-site
generator — each with its own robots.txt (or none at all, inheriting a platform default). It’s
easy for the documentation subdomain’s crawler policy to drift from what the main site intends,
simply because it’s a separately deployed artifact nobody is auditing as a pair.
Deployment and CDN drift
Static-site generators, documentation platforms, and CDNs can generate or modify robots.txt and
related headers automatically — a deploy-preview environment, a platform default, or a caching
layer can each independently affect what’s actually served, separate from what’s committed in a
repository. See the platform guides below for specifics on Vercel, Netlify, and Cloudflare.
The proof boundary
What CrawlPact does — and does not do
CrawlPact audits and monitors the public policy signals a website publishes. It does not control crawlers or guarantee that any of them comply.
What it does
- Checks the live, deployed response of each documentation and product domain
- Flags changes caused by your hosting platform, not only by your own commits
- Attaches the fetched evidence to every finding
What it does not do
- Improve search or AI-citation ranking automatically, or guarantee AI citation
- Manage or edit documentation content, or deploy policy fixes for you
- Index or store your documentation's page content — only its public policy signals
- Claim native, first-party integrations with documentation platforms
The platform guides describe what is independently verifiable about each platform's public behaviour — not a built integration.
Read the methodology for exactly what is evaluated, how the Policy Health Score is calculated, and the limitations before relying on a result.
Plan fit
Pro is the usual fit
Room for documentation, product and marketing domains, weekly monitoring, and domain groups to keep them together as one set.
Also consider Solo. Fits one documentation domain and a main site, rechecked monthly.
Every plan, including Free, gets the same complete, accurate audit. Paid plans add saved domains, monitoring, history, and export and sharing capacity — not a more accurate result.
| Feature | Solo | Pro |
|---|---|---|
| Monthly price | $9 | $19 |
| Yearly price | $89 | $189 |
| Saved domains | 5 | 25 |
| Automatic monitoring | Monthly | Weekly |
| Manual rescans per domain / month | 5 | 10 |
| Audit history | 12 months | 24 months |
| Private Atom change feed | Included | Included |
| Domain groups | Not included | Included |
| CSV domain export | Not included | Included |
| Batch domain import | Not included | Up to 10 |
| Agency-branded shared reports | Not included | Not included |
Questions
Frequently asked questions
Does CrawlPact see my documentation's actual content?
CrawlPact evaluates publicly accessible policy signals (robots.txt, meta directives, relevant headers) — it does not index or store your documentation's page content.
Will CrawlPact catch a change made by my hosting platform, not my own commit?
Yes — monitoring rechecks the live, deployed response, so a platform-side change is detected the same way a self-authored one is.
What plan fits a small documentation team with a couple of related domains?
Pro typically fits. The plan section above shows its current saved-domain allowance, monitoring cadence and domain groups, read live from the plan catalog.
Relevant platform guides
- AI crawler policy on CloudflareHow Cloudflare's managed robots.txt and AI Crawl Control interact with your site's public AI crawler policy, verified against Cloudflare documentation.
- AI crawler policy on NetlifyHow Netlify's deploy-preview noindex header, _headers file and deploy contexts change what your site tells AI crawlers, per Netlify's documentation.
- AI crawler policy on VercelHow Vercel's preview noindex header, static or generated robots.txt and custom headers change what your site tells AI crawlers, per official docs.
Related guides
- How to publish an llms.txt fileA step-by-step implementation guide for adding a valid llms.txt (and llms-full.txt) file to your site.
- My llms.txt file isn't validating — troubleshootingCommon structural issues CrawlPact's llms.txt validator flags, and how to fix each one.
- robots.txt vs. meta robots tag vs. X-Robots-Tag: which mechanism to useThree different mechanisms can restrict crawler access, at different scopes and for different content types. A decision guide for choosing between them.
Other teams CrawlPact serves
- AgenciesAudit, explain, and monitor AI crawler policy across every client website from one place — evidence-backed, without exposing internal tooling to clients.
- PublishersAudit and monitor the difference between what your site tells AI search crawlers and what it tells AI training crawlers — with evidence, not guesswork.
- Web developersConfirm the AI crawler policy you deployed matches what you intended, across frameworks, hosting platforms and CDNs that rewrite public files.
See what your own site tells AI crawlers
Run a free audit on any public domain. The report shows every finding with the evidence behind it — the same complete audit on every plan.