How to set Content Signals: the robots.txt directive and the Content-Signal header

Content Signals is an emerging convention for declaring simple, machine-readable preferences — such as whether AI training is allowed — as comma-separated key=value pairs. The Content Signals Policy (contentsignals.org) and Cloudflare document it as a Content-Signal line inside a robots.txt group. Some sites also send the same pairs as a Content-Signal HTTP response header, which neither documents.

Step 1: Decide the values you want to declare

The three purpose keys are search, ai-input and ai-train, each set to yes or no. A key you leave out neither grants nor restricts that use. For example, to allow search use but disallow AI training:

Content-Signal: search=yes, ai-train=no

Cloudflare is also testing an optional fourth field, use, which sets a ceiling on how much a crawler may keep and reuse: immediate, reference or full, from narrowest to broadest. It does not replace the purpose keys — state ai-train=no explicitly if that is your intent. As of 2 October 2026 contentsignals.org does not yet document use; see what immediate, reference and full mean before adding it.

Content-Signal: search=yes, ai-train=no, use=reference

Step 2: Add the directive to robots.txt

Put the line inside the robots.txt group it applies to — usually the User-agent: * group:

User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

A crawler-specific group can carry its own line if you want different preferences for one token. Content Signals sits alongside your Allow and Disallow rules; it does not replace them.

On Cloudflare, the managed robots.txt setting adds Content-signal: search=yes, ai-train=no, use=reference for you. Cloudflare announced on 15 September 2026 that Managed robots.txt is being deprecated in favour of Bot Preference Sync — check your live robots.txt after any change. See the Cloudflare platform guide.

Step 3 (optional): Send the same pairs as an HTTP response header

Neither contentsignals.org nor Cloudflare documents a header form, so treat it as a supplement to the robots.txt line, not a replacement — most sites do not need it. If you have a reason to send it, add it wherever your responses are generated:

• Netlify — a _headers file in your publish directory:

  /*
    Content-Signal: search=yes, ai-train=no

• Vercel — a headers entry in vercel.json:

  {
    "headers": [
      {
        "source": "/(.*)",
        "headers": [{ "key": "Content-Signal", "value": "search=yes, ai-train=no" }]
      }
    ]
  }

• Cloudflare Pages/Workers Assets — a _headers file at your project root, in the same path / Header: value format Netlify uses.
• Reverse proxy / web server (Nginx, Apache, Caddy) — a response header directive in your site configuration (e.g. Nginx’s add_header, Apache’s Header set).
• WordPress — a functions.php snippet (or a small plugin) hooked to the send_headers action, calling header('Content-Signal: search=yes, ai-train=no').
• Application framework — set the header in your application’s response-handling code.

Keep the header and the robots.txt line identical; two different statements of the same preference invite a crawler to pick either.

Step 4: Verify

Fetch https://yourdomain.com/robots.txt directly and confirm the Content-Signal line appears in the group you intended — this is the authoritative check for the documented form.

CrawlPact’s Content Signals checker reads the Content-Signal lines in your live robots.txt and shows search, ai-input and ai-train for each User-agent group, with line numbers. It warns about a line outside any group, a value other than yes or no, and a group that declares both yes and no for one signal. It reports use as Cloudflare’s testing-stage extension without interpreting it, and also shows a homepage Content-Signal header if you send one.

Keep it consistent with your other declarations

If you also publish an RSL licence, make sure the two agree — CrawlPact flags an RSL licence that does not permit AI training next to ai-train=yes, and an RSL licence that permits it next to ai-train=no, since a crawler reading only one of the two could reach either conclusion.

What this doesn’t do

Content Signals is a declaration, not technical enforcement — see /limitations. Cloudflare documents that its own Browser Run /crawl endpoint honours these signals; whether any other crawler does is up to its operator.