CrawlPact vs Known Agents: Policy Audit vs Live Agent Analytics

Short answer

CrawlPact and Known Agents look at the same website from opposite directions. CrawlPact asks “what has this site publicly declared about crawler access, and do those declarations agree?” Known Agents — the product formerly called Dark Visitors — asks “which agents, crawlers, scrapers and AI-referred visitors are actually showing up, and how should we track or manage them?”

Choose CrawlPact for independent evidence of declared policy across public signals. Choose Known Agents for live agent analytics, AI chat referral measurement, robots.txt rules that maintain themselves, or identification of the agent behind a request. Teams that need both keep the declaration and the observed traffic as two separate datasets.

What CrawlPact solves

CrawlPact never touches your traffic. It retrieves the public crawler-policy resources a site serves and reports what each one declares, matched against a source-backed registry of documented crawler tokens. That makes it a check on the website as deployed — useful because the published policy is often the sum of a CMS setting, a framework default, a CDN rule and a forgotten plugin.

It cannot tell you whether GPTBot or ClaudeBot visited yesterday. It can tell you whether the public site expresses a rule for that token, which rule and line match, whether another supported signal contradicts it, and — on paid plans — whether the declared policy changed between scheduled scans.

What Known Agents solves

Known Agents describes four products. Agent Analytics shows crawlers, scrapers and AI agents visiting a site, using server-side integrations so bots that never run JavaScript are still counted. AI Chat Referral Tracking measures human visits that arrive from ChatGPT, Claude, Gemini and other AI chat platforms. Automatic Robots.txt adds category-based rules to your robots.txt and keeps them current as new bots are catalogued. The Agent Identification API takes a request’s headers and returns who the agent is, its type and operator, and whether the claim could be verified.

The rename from Dark Visitors is recent and deliberate — the company says visitors heard “dark” as “bad” — so older references to Dark Visitors describe the same product line.

Where they overlap

Both depend on an accurate understanding of crawler tokens, and both touch robots.txt. The roles are different, though: Known Agents can write and maintain robots.txt rules and pair them with traffic data; CrawlPact reads whatever the live site ends up serving and reports it with evidence. A writer and an independent reader of the same file are complementary.

Key differences

Traffic evidence versus policy evidence

A Known Agents dashboard is evidence about visits. A CrawlPact report is evidence about declarations. A perfectly declared rule can still be ignored by a crawler; a crawler can also be absent from a short analytics window while the policy for it is missing or contradictory. Treating one dataset as proof of the other produces false confidence in either direction.

Generating a policy versus verifying one

Automatic Robots.txt reduces maintenance for teams that want to block or allow whole categories of bots without editing tokens by hand. CrawlPact does not modify robots.txt at all. Keeping the system that writes a policy separate from the system that checks the published result is a familiar control: it catches the cases where the generated block never reached production, was overridden by another layer, or conflicts with a header.

Request identity versus a crawler registry

CrawlPact’s registry interprets declared tokens against operators’ documentation; it never looks at an incoming request. Known Agents’ Identification API is built for exactly that request-level question, returning a verification result alongside the agent name. If you need to know whether a request claiming to be a known crawler is genuine, that is the tool shape you need. CrawlPact’s Web Bot Auth analysis explains why identity and permission are separate questions.

Installation and data access

CrawlPact audits any public domain with no code, connector or log export. Known Agents needs a platform connector or backend integration, because observed traffic can only come from where requests land. That makes it operationally richer for analytics and CrawlPact lighter for external review.

What Known Agents does that CrawlPact does not

• Observed agent traffic — which crawlers, scrapers and AI agents visited, and what they requested.
• AI chat referral measurement — human visits attributed to AI chat platforms. That is a growth and attribution question CrawlPact deliberately does not answer.
• Self-maintaining robots.txt rules by bot category.
• Request-level identification and verification through an API you call from your own code.

Choose CrawlPact when…

• You need an independent review of what the live site publicly declares.
• You want robots.txt, meta and header directives, llms.txt, RSL and Content Signals checked for consistency, not just robots.txt generated.
• You manage domains across different hosts and CDNs and want one audit model.
• You need policy evidence and change history without sending traffic logs anywhere.
• You want the verifier to be independent of whatever generates or enforces the policy.

Choose Known Agents when…

• You need to see which AI agents, crawlers and scrapers are visiting.
• You want to measure human traffic referred by AI chat platforms.
• You want category-based robots.txt rules kept current as new bots appear.
• You need to identify or verify the agent behind incoming requests from your own code.
• You want policy decisions informed directly by observed agent activity.

Use both when…

The combined loop is observe, decide, publish, verify. Known Agents shows which agents actually interact with the property and can maintain the robots.txt rules you choose. CrawlPact then confirms that the public site serves the intended policy after the change. If analytics show a crawler still arriving despite a declared restriction, you hold two separate pieces of evidence — the declaration and the request pattern — and can move to enforcement without confusing one for the other.

Important limitations

• CrawlPact cannot tell you which bots visited or whether a request is genuine.
• Known Agents’ analytics coverage depends on the integration and on what data each platform exposes.
• Automatic Robots.txt is still a declaration; whether a bot complies depends on the bot or on a separate enforcement layer, as Known Agents’ own documentation says.
• Known Agents was previously called Dark Visitors; this page uses the current name.
• Product details change. The facts here were verified against Known Agents’ documentation on the date shown.