CrawlPact

Troubleshooting guide

My RSL or Content Signals declaration isn't being detected — troubleshooting

Common reasons CrawlPact reports no RSL licence or no Content Signals for a site that publishes them, and how to check each one.

By CrawlPactPublished Updated

Both declarations only count if they are served where clients look for them. This guide covers the checks CrawlPact has run since ruleset 2026.10.1 (15 October 2026). A report from an earlier scan says so and shows the result as it was originally evaluated — run a new audit to apply the current checks.

Why does CrawlPact say no RSL licence is declared?

CrawlPact follows RSL 1.0’s discovery rules. It looks for a License: line in your robots.txt, a Link header with rel="license" and type="application/rsl+xml" on your homepage response, and a matching <link> or inline <script> in your homepage HTML. Only if none exists does it try the legacy /.well-known/rsl.xml location. Common causes:

  • No association. The XML file exists but nothing points to it. Add a License: https://yourdomain.com/license.xml line to robots.txt (see how to publish an RSL declaration).
  • A relative URL. RSL 1.0 requires an absolute URI in License: lines and HTML links; License: /license.xml is ignored and reported as a warning.
  • The association is scoped to one crawler. A License line inside a named crawler’s group applies only to that crawler. CrawlPact lists it but evaluates the licence that applies to crawlers in general (the User-agent: * group, or a line before any group).
  • The link is missing its type. A <link rel="license"> without type="application/rsl+xml" could be any licence (Creative Commons, for example) and is not treated as RSL.

Why is my RSL licence reported as not conformant?

The licence was found but does not meet RSL 1.0. The report lists each error with its line. The usual ones:

  • The <rsl> root has no xmlns="https://rslstandard.org/rsl", or uses a different (earlier draft) namespace.
  • <permits> or <prohibits> has no type attribute, or uses words outside the vocabulary.
  • A <content> element has no url, or no <license>.
  • The file is not well-formed XML, or includes a DOCTYPE — CrawlPact does not process DTDs.

Why does CrawlPact say there are no Content Signals?

CrawlPact reads Content-Signal lines in your live robots.txt, the form Cloudflare documents. Common causes:

  • The line is outside a group. A Content-Signal line before the first User-agent line applies to no crawler; move it into the User-agent: * group (or the group it is meant for).
  • The line sits at the end of the file. A Content-Signal line placed after the last group’s rules — even after a blank line or a comment — belongs to that last group under RFC 9309 grouping, so it may apply to one crawler rather than all of them. CrawlPact warns about this; put the line inside the User-agent: * group instead.
  • The values are not yes or no. ai-train=false or search=maybe is reported as a warning and treated as unspecified.
  • Your CDN or platform serves a different robots.txt. Fetch https://yourdomain.com/robots.txt and confirm the line is in the live file — a managed or generated robots.txt can replace the one in your repository.
  • You only send a response header. Cloudflare documents Content Signals in robots.txt, not as a header. CrawlPact still shows a homepage Content-Signal header if you send one, but readers of Content Signals look in robots.txt.

A robots.txt without any Content-Signal line is not an error: it simply means the site neither grants nor restricts those uses through Content Signals.

Verify

Run CrawlPact’s RSL validator or Content Signals checker after making a change — both fetch your live site on every scan.

Related guides

See how this applies to your own site

Run a free audit to check your declared AI crawler policy against your own domain.

Analytics preferences

CrawlPact uses optional Google Analytics and Microsoft Clarity on public pages to learn which content is useful. Clarity records clicks and scrolling (session replay), with anything you type masked. Neither runs in the app or admin areas, and CrawlPact works the same whether you accept or decline. See the privacy policy.