CCBot

Questions about CCBot

What is CCBot?

CCBot is the crawler of Common Crawl, a non-profit that provides an open repository of web crawl data that anyone can access and analyse.

How do I block CCBot?

Common Crawl's own example is a group for User-agent: CCBot with Disallow: /. It applies to future crawls; CrawlPact cannot tell what is already in Common Crawl's archive.

CCBot is operated by the non-profit Common Crawl Foundation to build a large, openly available web crawl dataset.

Why it matters for AI policy specifically

Common Crawl’s dataset is widely reused as a training source by third parties beyond Common Crawl itself. A website that blocks every named AI-training crawler but leaves CCBot unaddressed may still have its content indirectly available for training through downstream reuse of the Common Crawl corpus — a conflict CrawlPact surfaces explicitly rather than silently.

Site-owner controls

Disallowing CCBot removes this content from future Common Crawl dataset snapshots going forward. It has no effect on any downstream party that already holds an earlier snapshot, and no effect on any other operator’s crawler evaluated separately by CrawlPact (such as GPTBot or ClaudeBot) — CCBot governs only Common Crawl’s own future crawling, not a broader AI-training relationship. See /limitations for what a robots.txt rule can and cannot guarantee.

How to verify CCBot requests

Common Crawl Foundation documents this user agent:

                  CCBot/2.0 (https://commoncrawl.org/faq/)
                

Common Crawl says CCBot requests resolve, in both directions, to hostnames under crawl.commoncrawl.org, and publishes its IPv4 and IPv6 ranges at https://index.commoncrawl.org/ccbot.json. Operator documentation.

A user-agent string alone can be copied by anyone; confirming a request really came from Common Crawl Foundation needs your server or CDN logs. CrawlPact audits declared policy and does not see your traffic.

Example robots.txt configuration

To disallow CCBot specifically, without affecting any other crawler:

User-agent: CCBot
Disallow: /

If no dedicated User-agent: CCBot group exists in a domain'srobots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works. A robots.txt rule is a published instruction; it does not technically prevent access.

Official source

https://commoncrawl.org/ccbot

Verified against the source above as of 1 July 2026 — see how CrawlPact verifies crawler information.