CrawlPactUse cases
AI crawler policy by team
The same audit applies to every website, but what goes wrong — and who has to notice — differs by team. Pick the page closest to how you work.Agencies
AI crawler policy for agencies
Agencies must understand, explain, and monitor AI crawler policy state across many client websites at once — one-off manual checks don't scale, and clients ask about it in language that doesn't map cleanly to a robots.txt file.
- Policy drifts between client check-ins
- You hear about it from the client
- Clients ask in business language
Publishers
AI crawler policy for publishers
Search and AI-training crawlers often warrant different decisions, but a site's public signals frequently don't distinguish them clearly — and CDN or deployment changes can alter the actual response without an editorial decision being made.
- One rule for “AI bots generally”
- Signals that contradict each other
- A platform quietly rewrites the file
SaaS and documentation teams
AI crawler policy for SaaS and documentation teams
Documentation discoverability decisions, the search-versus-training distinction, and deployment-platform-driven policy drift all intersect on the domains SaaS teams run — often across multiple subdomains with different owners.
- Docs live on their own deployment
- Subdomains drift apart
- Platforms change what's served
Web developers
Verify AI crawler policy after every deployment
The crawler policy a developer intends to ship and the crawler policy a framework, hosting platform, or CDN actually serves can diverge — and that gap is invisible until someone checks the live, deployed response.
- Build output isn't your source
- Preview copies production
- A stale cache after deploy
Every team gets the same complete audit for free. Ongoing checks are explained on the monitoring page, and platform-specific setup lives in the platform guides.