AI Crawler Access: Is Your Site Even Readable by ChatGPT?
If AI bots can't crawl your site, nothing else matters. robots.txt rules, JavaScript rendering risks, and how to make your content machine-accessible.
Why this is the first thing to check
Every other improvement depends on an AI crawler being able to read the page. If access is blocked, or the content only exists after JavaScript runs, then schema, content structure, citations and reviews all become invisible at the moment that matters. This is the cheapest and most decisive check in an AEO programme: it takes minutes, it has a binary answer, and if it fails nothing else you do will show up.
robots.txt: what it does and what it cannot do
robots.txt is a plain text file at the root of your domain that tells automated crawlers which paths they may request. It is a request, not a security control — well-behaved crawlers honour it, others ignore it, so it should never be used to hide anything genuinely private. The mistake that matters here is a blanket block: a robots.txt containing "User-agent: *" followed by "Disallow: /" tells every crawler to stay away from the whole site. This is usually left over from a staging build and is sometimes live for months without anyone noticing, because normal visitors and search engines that cached the site earlier keep working.
The AI crawlers, by name
Different assistants crawl under different names, so a rule that blocks one may leave another working. The main ones you will see in logs are GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, and Google-Extended, which controls whether Google's generative products may use your content. There is also a long tail of smaller agents. Do not guess at these names — read the provider's current documentation, because they change, and confirm against your own server logs rather than a third-party list that may be out of date.
The deliberate block, and when it is correct
Some businesses choose to block AI crawlers, and that is a legitimate position. What is not legitimate is doing it accidentally. If you do intend to block, do it explicitly, per agent, and document why — so that a future developer does not "fix" it as an oversight. If you intend to be found in AI answers, which is the premise of this whole programme, then the correct configuration is to allow the assistant crawlers you want to be cited by, and to check that your allow rules are not contradicted by a broader disallow later in the file. Later rules can override earlier ones depending on how they are written, so read the file top to bottom rather than assuming the first line wins.
The JavaScript rendering trap
This is the fault that costs the most and is hardest to see. If your site is built as a client-side application, the initial HTML may contain almost no content — just a shell and a script tag — with the actual text, headings and schema injected by JavaScript after load. Traditional search engines often render JavaScript, so the site appears healthy in Search Console. Many AI crawlers do not render it at all, or render only a limited subset. The result is a site that looks fine to you and to Google, and is effectively empty to the assistants you are trying to be cited by.
How to test whether your content is really there
Do not use the browser's element inspector. It shows the live DOM after JavaScript has run, which is precisely the thing you are trying to rule out. Instead view the page source, or fetch the URL with a plain HTTP request and no JavaScript engine. Then search that raw response for a sentence you know is on the page. If the sentence is absent, the content is client-rendered and at risk. Repeat for your main headings, your key facts such as service area and phone number, and your JSON-LD block. Server-side rendering, static generation, or prerendering for crawlers all solve this; which one is right depends on your stack.
Other access faults worth ruling out
A robots meta tag with noindex on a page you want indexed. A firewall or bot-protection layer that challenges automated requests — this is common on hosts with aggressive WAF settings and it can block legitimate crawlers while letting normal visitors through. A sitemap that lists URLs that redirect, so the crawler follows a chain and gives up. Pages that return a 200 status but render an error message. An llms.txt or robots.txt that is served with the wrong content type. Each of these is quick to check and each one can silently zero out the rest of the work.
The check to run every quarter
Access is not a one-time fix, because hosting configurations, security plugins and site rebuilds change it. Re-fetch your three most important pages as plain HTML each quarter and confirm the content and the schema are both present. Confirm robots.txt still allows the crawlers you want. Check your server logs for the assistant user agents you expect — their absence is a signal in itself, and it is usually the first sign that something upstream started blocking them.
Don't want to DIY?
Crawler access auditing, robots.txt policy, rendering checks and llms.txt publication are part of every Click Elevate plan. We test with plain HTTP requests rather than a browser, because that is what actually reveals whether the content is reachable.
This guide resolves these report findings
- Site reachable by crawlers
- robots.txt AI-bot policy
- JavaScript-rendered content risk
Run the free AEO score to see which of these apply to your site.
Did this guide answer your question?
See how visible your business is to AI
Get your free 0–100 AEO score across ChatGPT, Perplexity, Gemini, Claude, and Google AI Overviews.