CrawlPactPlatform guides
How AI crawler policy signals — robots.txt, meta directives, and response headers — actually work on specific hosting platforms and CDNs. Platform behaviour can change; every guide below shows when its official sources were last verified, and links directly to them.
Managed CDN
Cloudflare
How Cloudflare's managed robots.txt and AI Crawl Control interact with your site's public AI crawler policy — verified against official Cloudflare documentation.
Hosted application platforms
Shopify
How Shopify's robots.txt.liquid customization mechanism works, its default rules, and its documented limits — verified against official Shopify documentation.
WordPress
How WordPress's virtual robots.txt, the 'Discourage search engines' setting, and theme/plugin interactions affect what your site tells AI crawlers — verified against official WordPress documentation.
Deployment platforms
Netlify
How Netlify's automatic deploy-preview noindex header, _headers file, and deploy contexts affect what your site tells AI crawlers — verified against official Netlify documentation.
Vercel
How Vercel's automatic preview-deployment noindex header, static and generated robots.txt files, and custom headers affect what your site tells AI crawlers — verified against official Vercel and Next.js documentation.
Check your own deployed policy
Run a free audit to see exactly what your domain currently tells AI crawlers, regardless of which platform serves it.
Audit a domain