Scanner information

Current status

The live scanner is enabled in this environment. Submitting the audit form runs a real scan against the target's declared AI crawler policy, following the behaviour and safety boundaries described below. See status for the current development phase.

What the scanner does

• Fetches only public HTTP/HTTPS resources you specify by domain or URL.
• Identifies itself with a fixed user agent: CrawlPactAuditBot/1.0 (+https://crawlpact.com/scanner).
• Uses roughly 12 external requests per scan, each with connection, first-byte, and total-scan timeouts, and each response capped at roughly 2 MB.
• Follows at most five redirects, revalidating each destination before following it.
• If a target blocks, rate-limits, or otherwise refuses the scanner, that refusal is reported as a finding — not silently ignored, and never presented as if the resource were absent.

What the scanner does not scan

• Authentication-gated pages, or anything behind a login.
• Content that only appears after client-side JavaScript runs — the scanner reads the raw HTTP response, it does not render a page in a browser.
• An entire website — sitemap-listed URLs are sampled within a bounded limit, not crawled exhaustively.

Private, authenticated, and internal systems are excluded by design, not by omission — a public policy audit has no legitimate reason to reach them. See privacy for how long retrieved evidence is kept.

What the scanner never does

• Execute JavaScript on the target page.
• Submit forms or authenticate as a user.
• Retain cookies from the target website.
• Impersonate a third-party crawler's user agent.
• Fetch private, loopback, link-local, reserved, or cloud-metadata addresses, or a literal IP target.

Safety design

Every outbound request the scanner makes goes through a single, isolated safe-fetch module responsible for target validation — see security and the architecture decision record for scanner isolation.