CrawlPactResources
Platform guides
How robots.txt, meta directives and response headers actually behave on specific hosting platforms and CDNs — every guide cites official sources and shows when they were last verified.
Managed CDN
Cloudflare
Cloudflare sits in front of many sites as a CDN/edge layer and offers two AI-crawler-relevant features: an optional managed robots.txt that adds AI-crawler-blocking rules, and AI Crawl Control, a monitoring/permission layer for AI crawler access.
Hosted application platforms
Shopify
Shopify serves a default robots.txt optimised for SEO and allows merchants to customise it via a robots.txt.liquid theme template with a deliberately limited set of Liquid objects — customisation is explicitly unsupported by Shopify Support.
WordPress
WordPress core generates a virtual robots.txt when no physical file exists, and its 'Discourage search engines from indexing this site' setting behaves differently depending on WordPress version — a meta tag today, a robots.txt change in versions before 5.3.
Deployment platforms
Netlify
Netlify automatically applies X-Robots-Tag: noindex to Deploy Previews, unpublished production deploys, and old branch deploys — only the currently published production deploy and the most recent branch deploys are indexable by default.
Vercel
Vercel automatically adds an X-Robots-Tag: noindex header to preview and outdated production deployments, with a documented custom-domain exception — separate from whatever robots.txt your framework serves for production.
Check your own deployed policy
Run a free audit to see exactly what your domain currently tells AI crawlers, regardless of which platform serves it.