CrawlPactFor Agencies
AI crawler policy for agencies
Agencies must understand, explain, and monitor AI crawler policy state across many client websites at once — one-off manual checks don't scale, and clients ask about it in language that doesn't map cleanly to a robots.txt file.
The audit is free — no account and nothing to install.
Agency plan
$39 / monthor $389 / year
- 100 saved domains
- Weekly automatic monitoring
- 36 months of audit history
The problem
Where it usually goes wrong
Policy drifts between client check-ins
Hosting changes, theme updates and CDN migrations all touch robots.txt and related public signals without anyone deciding to change AI crawler access.
You hear about it from the client
Drift usually surfaces when a client asks why a crawler they expected to allow is blocked — or always was — after the fact, not before.
Clients ask in business language
A client's question about an AI product rarely maps cleanly to a robots.txt group. You need evidence you can show, not a conclusion you assert.
The registry moves too
A newly documented crawler token, or a corrected purpose classification, can change what a client's unchanged policy now means.
The workflow
How CrawlPact fits the work
Audit, keep the evidence, monitor for change. Every decision about what to change stays with you.
Step 1: Audit each client domain
A full, source-cited evaluation of what the site tells search, training, retrieval and agent crawlers — run the same way on every plan.
Step 2: Establish a baseline
The audited state becomes the reference point for future change detection.
Step 3: Find conflicts and gaps
Where a robots.txt rule, meta directive and header disagree, or where a crawler's access isn't addressed at all.
Step 4: Apply client-approved changes
CrawlPact surfaces findings and evidence. What to change is always the client's and agency's decision — never an automated action.
Step 5: Monitor the portfolio
Scheduled rechecks detect a change to the website's own configuration and a change to CrawlPact's verified crawler registry.
Step 6: Share or export evidence
A private, revocable report link — agency-branded on the Agency plan — or a CSV export for a client deliverable.
In depth
Why this is different for an agency
A single client’s crawler policy is a one-time question. A portfolio of client sites is a
recurring one — hosting changes, theme updates, and CDN migrations all touch robots.txt and
related public signals without anyone necessarily deciding to change AI crawler access. An agency
finds out about a policy drift when a client asks why their site suddenly (or has always) blocked
a crawler they expected to allow, or vice versa — usually after the fact, not before.
Client communication and evidence
Every report includes the evidence CrawlPact based its findings on — the actual fetched
robots.txt/meta/header content, not just a pass/fail summary — so an agency can show a client
exactly what was found, not just assert a conclusion. Reports can be shared via a private,
revocable link (never publicly indexable) and, on the Agency plan, carry the agency’s own name and
logo — CrawlPact’s own methodology and limitations disclosure always remains visible on a shared
report; branding never replaces or hides it.
Monitoring and registry changes
A monitoring cycle checks for two distinct kinds of change: the website’s own published policy changing, and CrawlPact’s own verified crawler registry changing (a new crawler being added, or an existing one’s classification being corrected against its operator’s documentation) — either can affect what a previously-audited site is now effectively telling crawlers, even with zero changes on the client’s side. Domain groups let you organise saved domains by client rather than one flat list, and batch import onboards a portfolio in one step.
The proof boundary
What CrawlPact does — and does not do
CrawlPact audits and monitors the public policy signals a website publishes. It does not control crawlers or guarantee that any of them comply.
What it does
- Reads the public robots.txt, meta and header signals each client site publishes
- Attaches the fetched evidence to every finding, so you can show a client exactly what was found
- Rechecks saved domains on a schedule and flags what changed — on the site or in the registry
What it does not do
- Change a client's website — applying a recommendation is always a manual step
- Provide a client-facing portal, per-seat team roles or an automated cross-domain comparison dashboard
- Verify domain ownership — an agency account is a single login managing its own saved domains
- Monitor daily or in real time — the fastest automatic cadence on any plan is weekly
Agency branding never replaces or hides CrawlPact's methodology and limitations disclosure on a shared report — a report's credibility depends on it being there.
Read the methodology for exactly what is evaluated, how the Policy Health Score is calculated, and the limitations before relying on a result.
Plan fit
Agency is the usual fit
Built for a client portfolio: the largest saved-domain allowance, weekly monitoring, batch import for onboarding, and the only plan with agency-branded shared reports.
Also consider Pro. Fits a smaller set of client domains that don't need agency-branded reports.
Every plan, including Free, gets the same complete, accurate audit. Paid plans add saved domains, monitoring, history, and export and sharing capacity — not a more accurate result.
| Feature | Pro | Agency |
|---|---|---|
| Monthly price | $19 | $39 |
| Yearly price | $189 | $389 |
| Saved domains | 25 | 100 |
| Automatic monitoring | Weekly | Weekly |
| Manual rescans per domain / month | 10 | 20 |
| Audit history | 24 months | 36 months |
| Private Atom change feed | Included | Included |
| Domain groups | Included | Included |
| CSV domain export | Included | Included |
| Batch domain import | Up to 10 | Up to 100 |
| Agency-branded shared reports | Not included | Included |
Questions
Frequently asked questions
Does CrawlPact make changes to a client's website?
No. CrawlPact reads publicly accessible policy signals and reports on them; applying any recommended change is always a separate, manual step the agency or client takes.
Can I white-label the entire product for clients?
Agency-plan shared reports carry your agency's name and logo. CrawlPact's own methodology and limitations disclosure stays visible on every shared report — it is not removable, since a report's credibility depends on that disclosure being present.
How many client domains can I monitor?
The plan section above shows the current saved-domain allowance and monitoring cadence for the Agency and Pro plans, read live from the plan catalog.
Relevant platform guides
- AI crawler policy on CloudflareHow Cloudflare's managed robots.txt and AI Crawl Control interact with your site's public AI crawler policy, verified against Cloudflare documentation.
- AI crawler policy on NetlifyHow Netlify's deploy-preview noindex header, _headers file and deploy contexts change what your site tells AI crawlers, per Netlify's documentation.
- AI crawler policy on ShopifyHow Shopify's robots.txt.liquid customization mechanism works, its default rules, and its documented limits — verified against official Shopify documentation.
- AI crawler policy on VercelHow Vercel's preview noindex header, static or generated robots.txt and custom headers change what your site tells AI crawlers, per official docs.
- AI crawler policy on WordPressHow WordPress's virtual robots.txt, the 'Discourage search engines' setting and plugins change what your site tells AI crawlers, per WordPress docs.
Related guides
- Blocking AI training while staying visible in AI searchChoosing between CrawlPact's presets when the goal is opting out of model training without losing AI-search discoverability.
- How to block only AI training crawlers, without blocking AI searchA step-by-step implementation guide for writing robots.txt groups that restrict training-purpose crawlers while leaving search-purpose crawlers untouched.
- My robots.txt rule isn't blocking a crawler — troubleshootingCommon causes when a Disallow rule doesn't appear to take effect against a specific AI crawler, and how to check each one.
Other teams CrawlPact serves
- PublishersAudit and monitor the difference between what your site tells AI search crawlers and what it tells AI training crawlers — with evidence, not guesswork.
- SaaS and documentation teamsAudit and monitor AI crawler access to product docs, knowledge bases and marketing domains — including what deployment platforms change for you.
- Web developersConfirm the AI crawler policy you deployed matches what you intended, across frameworks, hosting platforms and CDNs that rewrite public files.
See what your own site tells AI crawlers
Run a free audit on any public domain. The report shows every finding with the evidence behind it — the same complete audit on every plan.