CrawlPact
Cloudflare Content Signals use=reference: What immediate, reference and full Mean
Cloudflare's Content Signals use field caps reuse at immediate, reference or full, separate from search, ai-input and ai-train. What it means, and its limits.
AI crawler policy audit and monitoring
Audit and monitor your website's AI crawler policy.
See how search, training, user-triggered and agent crawlers are addressed by your website. Detect conflicts, get clear recommendations and monitor future changes — without installing anything.
No installation. No server-log access. No AI API required. Try example.com.
Why the results can be trusted
Vendor-neutral
Works with any host or CDN — no website has to move to audit it.
Evidence for every finding
The resource, line and rule behind each interpretation.
22 crawler reference pages
Covering 8 operators, each with an official source and a verification date. Full registry counts are in the Observatory.
Deterministic
Same input, same result. No AI model is used to judge your policy.
Why it matters
AI crawler policy is no longer a single allow-or-block decision.
Operators increasingly run separate crawlers for separate purposes. Blocking a training crawler is a different decision from blocking a search crawler — and a broad rule written for one can quietly catch the other.
Where policies usually go wrong
- Search blocked by accident. A broad AI-crawler rule also matches crawlers used for search or user-requested retrieval.
- Training left unspecified. The intent is to restrict training, but several documented tokens have no explicit rule. Unspecified is not the same as blocked.
- The public response drifts. A deployment, plugin or CDN rewrite changes what crawlers actually receive.
Not every website needs the same policy, and not all AI crawling is unwanted — the point is knowing, with evidence, what your website currently declares.
Search
Index pages so they can appear in search and AI answer experiences.
Training
Collect content that may be used to develop or train models.
User-triggered retrieval
Fetch a specific page because a person asked a product to, in real time.
Agents
Act on a page as part of an automated or semi-automated task.
5 further documented crawlers are classified as advertising and validation, research, mixed or unknown purposes. Every entry links to its operator's own documentation — see the AI crawler directory.
The proof boundary
What an audit can show — and what it cannot.
Every CrawlPact result keeps three things apart, so a report never claims more than its evidence supports.
What your website asks crawlers to do
robots.txt groups, robots meta, X-Robots-Tag, RSL and Content Signals, read exactly as published.
What CrawlPact actually retrieved
Status codes, redirects, content types and the exact lines matched, with retrieval timestamps.
Whether a crawler visits or obeys
Proving this needs your server or CDN logs. A public audit cannot see it, so CrawlPact never claims it.
Read the full limitations before relying on a result for a business decision.
How CrawlPact works
From a domain name to an evidence-backed report in one request.
- 01
Enter a domain
CrawlPact retrieves bounded public policy resources — nothing to install, no log access.
- 02
Review crawler access
Search, training, user-triggered and agent crawlers are evaluated and shown separately.
- 03
Fix conflicts
Each finding carries its evidence, an explanation and a copyable recommendation.
- 04
Monitor changes
Paid plans recheck on a schedule and record website and registry changes.
Sample report · synthetic data
See what a report looks like
Illustrative example using a synthetic demonstration domain — not a real scan result. Every finding follows the same order: what happened, why it matters, the evidence, the action, and the limitation.
Sample report — synthetic domain, not a real scan
68/ 100Needs attention
- Resource availability90
- Syntax & evaluation85
- Objective alignment45
- Cross-signal consistency70
sample-domain.example (demonstration data)
Completed with warnings- 1 Aug 2026, 09:00
- 2026.07.3
- 2026.07.2
- Allow search, block training
| Crawler | Purpose | Declared result | Matched rule |
|---|---|---|---|
| OAI-SearchBot | Search | Allowed | Allow: / (line 4) |
| GPTBot | Training | No explicit rule | — |
| ClaudeBot | Training | Blocked | Disallow: / (line 12) |
| Google-Extended | Training | Blocked | Disallow: / (line 12) |
| ChatGPT-User | User-triggered | Allowed | Allow: / (line 4) |
Medium severity
Example finding: GPTBot has no explicit rule
robots.txt does not name GPTBot in any User-agent group, so no explicit rule currently applies to it.
The selected objective ("Allow search, block training") intends to restrict model-training crawlers, but an unaddressed crawler is not the same as an explicit block.
robots.txt, fetched 2026-08-01 — no User-agent group names GPTBot.
Add an explicit Disallow rule naming GPTBot's user-agent token.
CrawlPact cannot confirm whether GPTBot actually requests this website — only that no rule currently addresses it.
Monitoring
A correct policy today can become outdated tomorrow.
Saved domains are rechecked on a schedule, and every change is attributed to its cause — your website, or what CrawlPact knows about crawlers.
Website-policy changes
- robots.txt, header or meta directive changed
- A resource became unavailable
- A CDN or plugin rewrote the public response
- A deployment changed the effective policy
Registry-driven changes
- A new crawler token was documented
- An operator changed a documented purpose
- Source verification changed
- Saved domains re-evaluated against the release
Monitoring frequency depends on your plan — see pricing. Changes surface in an in-app notification centre and a private Atom feed.
First scan establishes the declared policy
A deployment rewrites robots.txt
An operator documents a new crawler token
Scheduled scan confirms the current state
For agencies and multi-site teams
Govern crawler policy across every client website.
Save every client domain in one place, apply objective-based recommendations, and monitor for website or registry changes — without exposing internal tooling to clients.
- 01Audit client domains
- 02Establish policy baselines
- 03Identify conflicts and unspecified rules
- 04Apply client-approved recommendations
- 05Monitor website and registry changes
- 06Share or export evidence
- Portfolio monitoring
- Group domains, batch-import a client list, and schedule recurring rechecks — plan limits apply.
- Client-ready sharing
- Share a report with your agency name and logo. Methodology and limitations always stay visible; branding never replaces them.
Guidance by role
- AgenciesAgencies must understand, explain, and monitor AI crawler policy state across many client websites at once — one-off manual checks don't scale, and clients ask about it in language that doesn't map cleanly to a robots.txt file.
- PublishersSearch and AI-training crawlers often warrant different decisions, but a site's public signals frequently don't distinguish them clearly — and CDN or deployment changes can alter the actual response without an editorial decision being made.
- SaaS and documentation teamsDocumentation discoverability decisions, the search-versus-training distinction, and deployment-platform-driven policy drift all intersect on the domains SaaS teams run — often across multiple subdomains with different owners.
- Web developersThe crawler policy a developer intends to ship and the crawler policy a framework, hosting platform, or CDN actually serves can diverge — and that gap is invisible until someone checks the live, deployed response.
Free tools
Check a single signal in seconds.
Each tool runs the same real audit and opens the report at the section you care about — no separate, simplified logic.
- AI crawler checkerSee which documented AI crawlers your robots.txt allows or blocks, by purpose, with the rule and line behind each.
- Robots.txt AI validatorValidate robots.txt groups, rules, sitemap lines and syntax problems by line, following RFC 9309.
- RSL validatorFind the RSL licence a site declares and validate it against RSL 1.0, with its usage and payment terms.
- llms.txt validatorValidate llms.txt and llms-full.txt structure and linked resources.
- Content Signals checkerRead robots.txt Content-Signal lines (search, ai-input, ai-train) per User-agent group, with conflicts.
Supported public signals
What an audit reads.
Primary signals (robots.txt, HTTP headers, HTML meta directives) are checked on every scan. Optional and emerging signals are checked when publicly available; support depth varies with each specification's maturity, and none is required for AI visibility.
- robots.txt
- AI crawler user-agent groups
- llms.txt
- llms-full.txt
- RSL
- Content Signals
- Robots meta
- X-Robots-Tag
- Sitemap declarations
- Relevant HTTP headers
How each signal is parsed and weighed is documented in the methodology.
Evidence, in the open
Independent evidence, not another hosting-provider control panel.
CrawlPact has no testimonials or logo wall to show you. It publishes the method, the sources, the failures and the limits instead — check any of them before you trust a result.
| Record | What it shows |
|---|---|
| Methodology | How every signal is retrieved, parsed and weighed |
| Crawler directory | Each crawler's operator, purpose, official source and verification date |
| Status | Live service status and the incident history, including past failures |
| Security | Scanner containment, data handling and how to report a vulnerability |
| About | Who operates CrawlPact and what it will and will not do |
| Corrections process | How to report a wrong classification, and how fixes are published |
Research and analysis
Latest Insights
Source-backed analysis of changes in AI crawlers, search reporting, and publisher controls — plus reproducible research from the public registry.
Cloudflare's Disallow AI Training Setting: Stay Discoverable in Search Without Allowing Training
Cloudflare's Disallow AI Training keeps accountable mixed-use crawlers such as Googlebot for search while refusing training. How it differs from Block.
Google Multimodal Search Reporting: What Lens, Circle to Search and Image Queries Reveal
Search Console now splits web multimodal searches (Lens, Circle to Search, image uploads) from text queries. What the data shows, and what it cannot prove.
Plans
Pricing
Monthly and yearly pricing. Billing is handled by Paddle; applicable taxes may be calculated during checkout.
Free
For a one-time check on a single site.
Scan a websiteSolo
For individual site owners tracking one project.
Start monitoringPro
For growing portfolios that need weekly monitoring.
Choose ProAgency
For agencies managing many client domains.
Choose Agency
FAQ
Frequently asked questions
Short, factual answers. The long versions live in the methodology and limitations.
Free public audit
Check your website's AI crawler policy now.
Run a free public audit. No account or installation required.
Results describe published policy signals and do not guarantee crawler behaviour.
View a sample report