AI crawler policy on Netlify

What CrawlPact can verify

CrawlPact fetches the live, public robots.txt, response headers (including X-Robots-Tag), and meta directives for whichever URL you save — a production domain, or a specific deploy-preview URL if you save that instead.

What CrawlPact cannot verify

CrawlPact cannot see your Netlify site dashboard, deploy contexts, or netlify.toml configuration directly — only the resulting public response for a given URL. It also cannot determine Netlify’s own internal deploy-context classification (production vs. branch deploy vs. deploy preview) beyond what the response headers for that specific URL actually show.

Where crawler policy may originate on Netlify

• Netlify’s automatic X-Robots-Tag: noindex header — applied to Deploy Previews, unpublished production deploys, and old (non-current) branch deploys, with no configuration required. Only the currently published production deploy and the most recent branch deploys are indexable by default.
• A _headers file in the site’s publish directory, or [[headers]] tables in netlify.toml — for custom response headers, including overriding the default indexability behaviour above.
• A robots.txt file in the publish directory (static, or generated by your build), served like any other static asset.
• netlify.toml build/deploy-context configuration — commonly used to generate an environment-aware robots.txt (e.g. different output for production vs. deploy-preview contexts) as part of the build step, since headers alone cannot easily branch content by context.

Public signals relevant to Netlify

robots.txt, the X-Robots-Tag response header (both Netlify’s own automatic behaviour and anything set explicitly), and meta robots tags your build output includes.

Common conflicts or failure modes

• Assuming a custom _headers/netlify.toml header rule can be scoped to one branch or context. Per Netlify’s own documentation, headers declared in the standard _headers file or a standard netlify.toml [[headers]] table are global across all deploys — branch- or context-specific headers require a workaround (a custom build step producing a different file per context), not a built-in scoping option.
• A _headers/netlify.toml rule not applying to Functions, Edge Functions, or proxied content. Netlify’s documentation is explicit that these headers apply only to files Netlify serves from its own backing store — a Function or Edge Function must set its own headers at the origin if you need X-Robots-Tag or similar there.
• Not realising the most recent branch deploy is indexable by default. Netlify’s automatic noindex behaviour explicitly exempts the current production deploy and the most recent branch deploys — a team assuming “everything except production is hidden” can be surprised that their newest branch deploy is not automatically noindexed; Netlify’s own documentation notes this can be further restricted with a custom header if desired.

How to inspect the current public response

Request the specific URL you care about directly and inspect both /robots.txt and the X-Robots-Tag response header — for a deploy preview or branch deploy, confirm whether Netlify’s automatic noindex header is present for that specific deploy’s current state.

How to implement or update policy safely

• For a fixed robots.txt across all environments: place a static file in the publish directory.
• For an environment-aware robots.txt (different rules for production vs. preview): generate the file as part of your build step, conditioned on Netlify’s own build-time context environment variables, since a single static file or a global _headers rule cannot branch by context on its own.
• To further restrict indexability of your most recent branch deploy beyond Netlify’s default behaviour: add a custom X-Robots-Tag header for that context via _headers or netlify.toml.

How to verify after deployment

Re-check the X-Robots-Tag header and robots.txt output for both your production domain and any specific branch/preview URL you care about — particularly after changing which branch is your production branch, since Netlify’s automatic indexability behaviour follows the deploy’s current status, not a fixed URL.

Monitoring and change detection

A saved CrawlPact domain is automatically rechecked on your plan’s schedule (Solo: monthly, Pro/Agency: weekly), catching a change to a static robots.txt file, a build-generated one, or header configuration — including one introduced by a netlify.toml change.

Platform-specific limitations

• CrawlPact cannot distinguish Netlify’s internal deploy-context classification (production vs. branch deploy vs. deploy preview) — it reports the headers/robots.txt actually returned by the URL you save, the same evidence a real crawler would use.
• CrawlPact cannot read your netlify.toml or build configuration directly.
• Whether a _headers/netlify.toml rule is being correctly scoped to a specific context (given the documented lack of native branch-scoping) is a build-configuration detail CrawlPact cannot introspect beyond observing the actual served output at audit time.

Related tools and crawler pages

See robots.txt vs. meta robots vs. X-Robots-Tag for how these signal layers relate, and the crawler directory for documented AI crawler tokens.

Frequently asked questions

Are all my Netlify deploy previews hidden from AI crawlers? Per Netlify’s documentation, Deploy Previews and old branch deploys get an automatic X-Robots-Tag: noindex header — but the current production deploy and the most recent branch deploy do not, by default. Check the actual header for the specific URL you care about.

Does CrawlPact deploy or configure anything on Netlify? No — CrawlPact audits the public response your deployment already produces; changing netlify.toml, _headers, or build configuration happens in your own project.