CrawlPact vs Cloudflare AI Crawl Control: Policy Audit vs Edge Control

Short answer

These two products are not substitutes. CrawlPact reads the crawler policy your website publishes — robots.txt groups, robots meta and X-Robots-Tag directives, llms.txt, RSL and the homepage Content-Signal header — and reports what that policy declares to each documented AI crawler. Cloudflare AI Crawl Control (formerly Cloudflare AI Audit) works on traffic: for a domain proxied through Cloudflare it shows which AI crawlers are requesting pages, flags requests for paths your robots.txt disallows, and can block a crawler at the edge or, in private beta, charge it per crawl.

Pick CrawlPact when the open question is “what is our site actually declaring, is it consistent, and did it change?” Pick Cloudflare when it is “which AI crawlers are hitting our Cloudflare zone, and what should the edge do with them?” Many teams want both answers, and the two products give them from opposite sides of the request.

What CrawlPact solves

CrawlPact is a policy auditor that stays outside the request path. It fetches a bounded set of public resources the way any crawler could, parses them, and matches them against a versioned registry of documented AI crawler tokens. Every finding carries its evidence: the robots.txt group and line that matched, the header or meta tag that applied, and the registry version used. See the methodology for the exact rules.

Because it only needs a public URL, the same audit works on a Cloudflare zone, a Vercel deployment, a WordPress host or a custom origin. That matters when a policy is produced by several layers at once — a CMS plugin writes robots.txt, the framework sets a header, a CDN rewrites another — and nobody is sure what the live site finally serves.

What Cloudflare AI Crawl Control solves

AI Crawl Control is a Cloudflare dashboard product for zones whose traffic passes through Cloudflare’s reverse proxy. Its documentation describes four jobs: showing which AI services access your content and how; letting you allow or block individual crawlers; tracking robots.txt compliance; and setting up Pay Per Crawl pricing. Blocking is implemented as a single WAF custom rule, so an unwanted request can be stopped before it reaches your origin.

Its robots.txt view is about behaviour, not syntax. A violations table lists AI crawlers that requested paths your robots.txt disallows, the directive they breached and how often — evidence CrawlPact cannot produce, because CrawlPact never sees those requests.

Where they overlap

Both products care about robots.txt and crawler identity. Cloudflare compares observed requests with your robots.txt; CrawlPact evaluates the robots.txt rules themselves and places them beside the site’s other public declarations. The overlap is useful rather than redundant: one tool answers “what did the site publish?”, the other “what reached the network, and did it respect that?”

Key differences

Declared policy versus observed traffic

CrawlPact can audit a domain with no access to logs, DNS or a dashboard, because its subject is the public declaration. Cloudflare’s crawler view requires the domain to be connected to Cloudflare and proxied through it; in exchange it sees real requests, which no public audit can.

Many signals versus one enforcement point

A site can publish signals that disagree — robots.txt allows a crawler while an RSL declaration or a noindex header says something different. Surfacing that disagreement is what CrawlPact is built for. Cloudflare’s strength is the other end: once you decide, the edge can act on every matching request.

Hosting-neutral versus Cloudflare-native

AI Crawl Control is naturally strongest for zones already on Cloudflare. CrawlPact evaluates the public response, so agencies and groups with mixed infrastructure can audit every property with one model — including the ones that will never move to Cloudflare.

Blocking can reach search

Cloudflare’s own documentation warns that setting search-engine crawlers to Block or Charge “may negatively impact your site’s SEO performance”. That is a reason to separate crawler purposes — search, training, user-triggered fetching — before switching on any broad AI-bot action. CrawlPact reports declared outcomes separately by purpose, which makes that review concrete; the guide to blocking AI training while staying visible in AI search walks through the trade-off.

What Cloudflare does that CrawlPact does not

• Live AI crawler telemetry for a proxied zone, broken down by crawler, operator, path, status code and hostname.
• Edge enforcement: a block is applied to real requests before they reach your origin.
• Behavioural robots.txt evidence: which crawlers actually requested disallowed paths.
• Commercial access: Pay Per Crawl lets you allow, charge or block selected crawlers. Cloudflare lists it as private beta, so availability should be checked for your account.

CrawlPact offers none of these. It cannot tell you whether a crawler visited, ignored a rule or paid.

Choose CrawlPact when…

• You need one independent policy audit across sites on different hosts and CDNs.
• You want to see what the live site publicly declares, not only what a dashboard is configured to enforce.
• You want robots.txt, meta and header directives, llms.txt, RSL and Content Signals read together, with conflicts surfaced.
• You need evidence-backed reports, and — on paid plans — scheduled re-audits and change history for a portfolio.
• You need to confirm that a deployment serves the policy you intended.

Choose Cloudflare AI Crawl Control when…

• Your domain is already proxied through Cloudflare and you need to see AI crawler requests.
• You need a block enforced at the edge, before requests reach your origin.
• You want evidence of which crawlers request paths your robots.txt disallows.
• You are evaluating Pay Per Crawl, or Cloudflare’s wider bot-management tools.
• Your problem is request handling rather than independent policy review.

Use both when…

A clean split is CrawlPact as the external policy-evidence layer and Cloudflare as the traffic layer. Audit the public policy before you change anything; let Cloudflare observe and enforce at the edge; audit again after the deployment. The split helps most when application code or a CMS owns the published policy while an infrastructure or security team owns the edge rules — each team gets evidence about its own layer, and drift between the two shows up as a disagreement you can see.

Important limitations

• CrawlPact cannot prove that any crawler obeyed a rule; it audits declared policy and the responses it retrieved. See what CrawlPact cannot prove.
• CrawlPact has no access to your Cloudflare account and cannot read dashboard or WAF settings.
• CrawlPact reads Content Signals only from the homepage Content-Signal response header, not from a Content-Signal line in robots.txt.
• Cloudflare’s analytics describe only traffic that passes through Cloudflare for the configured zone; they are not an audit of every public declaration the site makes.
• Plans, detection methods and Pay Per Crawl availability change. The facts on this page were verified against Cloudflare’s documentation on the date shown and will be rechecked on every material update.