CrawlPactCurrent status
The live scanner is enabled in this environment. Submitting the audit form runs a real scan against the target's declared AI crawler policy, following the behaviour and safety boundaries described below. See status for the current development phase.
What the scanner does
- Fetches only public HTTP/HTTPS resources you specify by domain or URL.
- Identifies itself with a fixed user agent:
CrawlPactAuditBot/1.0 (+https://crawlpact.com/scanner). - Uses roughly 12 external requests per scan, each with connection, first-byte, and total-scan timeouts, and each response capped at roughly 2 MB.
- Follows at most five redirects, revalidating each destination before following it.
- If a target blocks, rate-limits, or otherwise refuses the scanner, that refusal is reported as a finding — not silently ignored, and never presented as if the resource were absent.
What the scanner does not scan
- Authentication-gated pages, or anything behind a login.
- Content that only appears after client-side JavaScript runs — the scanner reads the raw HTTP response, it does not render a page in a browser.
- An entire website — sitemap-listed URLs are sampled within a bounded limit, not crawled exhaustively.
Private, authenticated, and internal systems are excluded by design, not by omission — a public policy audit has no legitimate reason to reach them. See privacy for how long retrieved evidence is kept.
What the scanner never does
- Execute JavaScript on the target page.
- Submit forms or authenticate as a user.
- Retain cookies from the target website.
- Impersonate a third-party crawler's user agent.
- Fetch private, loopback, link-local, reserved, or cloud-metadata addresses, or a literal IP target.
Safety design
Every outbound request the scanner makes goes through a single, isolated safe-fetch module responsible for target validation — see security and the architecture decision record for scanner isolation.