robots.txt vs. meta robots tag vs. X-Robots-Tag: which mechanism to use

Three separate, standard mechanisms can express a crawling or indexing restriction, and they operate at different scopes. Using the wrong one for the job is a common source of confusion.

The three mechanisms

• robots.txt — a single file at the domain root, declaring rules per crawler token, scoped by URL path pattern. It restricts crawling (whether a page is fetched at all).
• Meta robots tag — an HTML <meta name="robots" content="..."> tag on an individual page. It restricts indexing/following of that specific page, but only takes effect if the crawler is actually allowed to fetch the page in the first place (a page disallowed in robots.txt is never fetched, so its meta tag is never seen).
• X-Robots-Tag — an HTTP response header, equivalent to the meta robots tag but usable on any response type, including non-HTML files (PDFs, images) where a <meta> tag isn’t possible.

The decision

• Use robots.txt to prevent a crawler from fetching a section of your site at all — this is the only mechanism of the three that stops the request itself.
• Use the meta robots tag when you want a page to be fetched (so its content can still be evaluated) but not indexed or followed from — for example, a private-but-linked page.
• Use X-Robots-Tag for the same per-page control as the meta robots tag, but on a file type that can’t carry an HTML <meta> tag, or when you want to set the directive at the server/CDN level rather than in every page’s HTML.

Common mistake CrawlPact flags

Adding a meta robots or X-Robots-Tag “noindex” directive to a page that’s also disallowed in robots.txt for the same crawler is redundant, and can create a false sense of double protection — since the crawler never reaches the page to see the tag at all. CrawlPact’s Content Signals checker surfaces the meta robots tag, X-Robots-Tag header, and canonical URL together, alongside the robots.txt result, so you can see whether they actually agree.

A real-world example

CrawlPact hit a version of this on its own site in September 2026: a Disallow: /audit/ rule meant for audit-result pages also blocked the public /audit/ landing page — and hid the result pages’ own noindex from crawlers. The Bing Site Scan case study walks through the root cause, why the fix was noindex on crawlable pages rather than a narrower block, and the test that now guards against it.