CrawlPactEvery figure below is computed directly from CrawlPact's immutable registry release 2026.09.1, resolved from the same frozen release snapshot production evaluation and historical scan rendering use — never from a live, mutable table, and never from the static crawler directory pages.
- Data collected
- 2026-10-22T10:21:26.120Z
- Registry release
- 2026.09.1
- Checksum
- 9a4f14303d99f24d8ba513d2d158263dd48df3710427fa3ac00065607bb9ed2b
- Methodology
- policy-observatory-methodology-v1
Crawler and operator counts
| Tracked crawler records | 24 |
|---|---|
| Evaluation-eligible (active, deprecated, replaced) | 24 |
| Distinct operators | 9 |
Purpose distribution
Denominator: 24 tracked crawler records.
| Purpose | Records |
|---|---|
| advertising validation | 2 |
| agent | 2 |
| mixed | 2 |
| research | 1 |
| search | 7 |
| training | 5 |
| unknown | 1 |
| user triggered | 4 |
Lifecycle distribution
| Lifecycle status | Records |
|---|---|
| active | 24 |
Operators by purpose
| Operator | Purposes |
|---|---|
| Amazon | mixed, search, user triggered |
| Anthropic | search, training, user triggered |
| Apple | mixed, training |
| Common Crawl Foundation | research |
| agent, search, training, unknown | |
| Meta | advertising validation, agent, search, training |
| Microsoft | search |
| OpenAI | advertising validation, search, training, user triggered |
| Perplexity AI | search, user triggered |
This lists which purposes each operator currently documents separate crawlers for — it is not a transparency or trust ranking (no operator quality score exists).
Source-verification freshness
Denominator: 24 tracked crawler records.
| Records with a recorded verification date | 24 |
|---|---|
| Never verified | 0 |
| Median verification age (days) | 92 |
| Due for re-verification (evaluation-eligible, >180 days or never) | 0 of 24 |
Release history
2 published release(s). Each row compares that release to the one immediately before it — evidence-only and editorial-only changes never triggered customer re-evaluation (see methodology).
| Release | Published | Added | Removed | Purpose changes | Token changes | Evidence-only |
|---|---|---|---|---|---|---|
| 2026.09.1(active) | 20 Sept 2026 | 1 | 0 | 0 | 0 | 0 |
| 2026.07.3 | 28 Jul 2026 | 23 | 0 | 0 | 0 | 0 |
Limitations
- These figures describe CrawlPact's own crawler-identity registry, not a sample of the web — they do not measure how many websites allow or block any crawler.
- Purpose classification reflects each operator's published documentation as independently reverified by CrawlPact; it does not measure actual crawler behaviour.
- An operator may run undocumented or unverified crawlers not yet reflected in this registry.