CrawlPactMonitoring
Monitor AI crawler policy changes across your websites
A deployment, CDN setting or plugin update can quietly change what your site tells AI crawlers. CrawlPact rechecks every saved domain on a schedule, records each change with its evidence, and tells you whether your website changed or what CrawlPact knows about crawlers did.
The audit needs no account. Save the domain from your report to start monitoring it.
A one-time audit answers "what does my site declare today?"
Monitoring answers the next question: "has that changed, and why?"
One-time audit — every plan, Free included
- Every documented crawler evaluated against your robots.txt, by purpose
- Content Signals, RSL, llms.txt and page directives read and checked
- Findings with the exact line, evidence and a recommended fix
- A Policy Health Score for that moment
Monitoring — paid plans
- A saved baseline per domain
- Automatic rechecks (monthly or weekly, by plan) plus manual rescans
- A change timeline with the cause of each change
- Findings tracked as they appear, change and resolve
- Notifications and a private Atom feed
What each recheck reads
Each scheduled check runs the same bounded audit as the free tool, through the same safe-fetch scanner, and compares it with the domain's previous complete scan.
| Resource | What a change can mean |
|---|---|
| robots.txt | A crawler moved between allowed, blocked and unspecified; a Content-Signal or License line was added, removed or edited |
| RSL licence document | Declared usage, payment or licensing terms changed |
| llms.txt, llms-full.txt | The file appeared, disappeared or changed |
| Homepage meta robots, X-Robots-Tag | Page-level indexing or snippet directives changed |
| Homepage Content-Signal header | A header form of Content Signals was added, removed or changed |
Every change is attributed to its cause
Website change
A resource your site serves changed between two complete scans — for example a deployment rewrote robots.txt.
Registry-driven change
Your site did not change, but a new crawler registry release did — say, a newly documented crawler token — so the same policy now reads differently.
Incomplete check
One of the two scans could not retrieve your site, so nothing is compared and no policy change is claimed.
When both happened at once, the change is labelled as both. Findings carry a stable fingerprint, so the timeline shows which appeared, changed or were resolved.
What a recorded change looks like
User-agent: GPTBot
Disallow: /
User-agent: *
Content-Signal: search=yes, ai-train=no
Allow: /User-agent: *
Allow: /- GPTBot (training): blocked byDisallow: / on line 2 → allowed, because only the wildcard group's Allow: / (line 2) now applies.
- Content Signals:ai-train=no → unspecified.
- Finding appeared: A training crawler is not restricted (under the "allow search, block training" objective).
How you hear about it
In-app notifications for policy changes, registry-driven changes, resources that stop responding, and paused monitoring.
A private Atom feed (Solo, Pro, Agency) — add it to a feed reader or another tool that reads feeds.
A change timeline per domain, with scan history kept for your plan's retention period.
CrawlPact does not send email, SMS or chat alerts.
For agencies and multi-site teams
Save every client domain, organise them into groups, import them in batches, and export the portfolio as CSV where your plan includes it. On Agency, shared reports can carry your own branding. Each domain keeps its own baseline, timeline and findings.
AI crawler policy for agenciesMonitoring by plan
| Plan | Saved domains | Automatic rechecks | Manual rescans / domain / month | History | Atom feed |
|---|---|---|---|---|---|
| Free | 1 | No automatic monitoring | 2 | 30 days | — |
| Solo | 5 | Monthly | 5 | 12 months | Included |
| Pro | 25 | Weekly | 10 | 24 months | Included |
| Agency | 100 | Weekly | 20 | 36 months | Included |
Free keeps 1 saved domain with no automatic rechecks. Prices, taxes and billing intervals are on the pricing page.
What monitoring does not do
- It does not watch traffic. It reports declared policy, not which crawlers visited or obeyed.
- It is not real-time; a change is detected at the next scheduled check or manual rescan.
- It reads the homepage and the policy files above, not every page of the site.
- It never edits your site, robots.txt or CDN settings.
Full detail: limitationsand what the scanner requests.
Questions about monitoring
- What is AI crawler policy monitoring?
- It is a scheduled re-audit of the crawler policy a website publicly declares. CrawlPact rechecks each saved domain monthly or weekly, depending on the plan, compares the result with the previous scan, and records what changed, with evidence.
- Is monitoring real-time?
- No. Checks run on a schedule (monthly or weekly, depending on the plan), and you can start a manual rescan within your plan's monthly allowance. A change made between checks is detected at the next one.
- How is monitoring different from a one-time audit?
- Every plan, Free included, gets the same complete audit. Monitoring adds saved baselines, scheduled rechecks, change history and notifications, so you learn when the declared policy changes instead of re-running audits by hand.
- How will I find out about a change?
- Through the in-app notification centre and, on the Solo, Pro, Agency plans, a private Atom feed you can add to a feed reader or another tool. CrawlPact does not send email, SMS or chat alerts.
- Does CrawlPact tell me whether my website or the crawler registry changed?
- Yes. Each change is classified as a website change, a registry-driven change (new information about crawlers re-evaluated against your unchanged policy), both, or not comparable because a scan was incomplete.
- What happens if my site cannot be reached during a check?
- The check is recorded as incomplete rather than as a policy change. If a saved domain fails five scheduled checks in a row, monitoring for it pauses and you are notified; you can resume it.
- Does monitoring change my website or prove what crawlers do?
- No. CrawlPact only reads public policy files and the homepage. It never modifies your site, and it reports declared policy, not real crawler traffic or compliance.
Start with what your site declares today
Run a free audit, then save the domain from the report to turn it into a monitored baseline.