The complete technical checklist for making your website crawlable and indexable by AI engines — sitemap, meta tags, page speed, and more.
Sitemap: https://yourdomain.com/sitemap.xmlAdd a robots.txt that explicitly welcomes AI crawlers:
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
Every page needs a unique meta description (150-160 characters) that:
Target: LCP under 2.5 seconds
Your entire site must use HTTPS. AI engines and search engines penalize HTTP sites.
Over 60% of searches are on mobile. Test your site at search.google.com/test/mobile-friendly.
Was this article helpful?