RSL vs. Content Signals vs. robots.txt: which one wins when they disagree

robots.txt, RSL, and Content Signals can each express a preference relevant to AI crawlers — and none of them is aware of the other two. Nothing enforces agreement between them; that’s left entirely to the site owner. They also answer different questions: robots.txt rules are about crawling (may this bot fetch this path?), while RSL and Content Signals are about use (may fetched content be used for search, AI answers or training?).

The three declarations

• robots.txt — per-crawler-token allow/disallow rules, evaluated per URL path.
• RSL (Really Simple Licensing) — an XML licence document, found through a License: line in robots.txt, an HTTP Link header or HTML, declaring permitted and prohibited uses (for example, prohibiting ai-train while saying nothing about search) plus optional payment terms.
• Content Signals — a Content-Signal line inside a robots.txt group (the form the Content Signals Policy and Cloudflare document) with search, ai-input and ai-train set to yes or no, plus an optional use field that Cloudflare is testing.

No single mechanism “wins”

None of these three specifications defines the other two as authoritative. A crawler operator that only reads robots.txt will never see your RSL or Content Signals declaration, and vice versa. The practical implication is that all three need to say the same thing for your intent to reach every crawler that might check any one of them.

What CrawlPact checks

CrawlPact compares the two use declarations with each other: an RSL licence that does not permit AI training next to a Content-Signal that says ai-train=yes, or the reverse. It reads Content Signals from the User-agent: * group of robots.txt. It deliberately does not treat a robots.txt Allow as consent to training, or a Disallow as a licence term — crawling and use are different questions. It does not (and cannot) resolve a disagreement for you; it only surfaces it so you can fix the declaration that’s wrong.

The decision

If you’re publishing more than one of these three mechanisms, treat updating all of them together as a single change, not three independent edits. CrawlPact’s RSL validator and Content Signals checker show each declaration on its own, so you can compare them directly before publishing. All three are declarations, not licences; for how a licensing and paid-access platform differs from a policy audit, see CrawlPact vs TollBit.