Can AI Crawlers Read Your Site? A Practical Audit

A site cannot be cited for information an answer engine cannot retrieve or understand. Yet "the page loads in my browser" is not a sufficient accessibility test. A crawler may encounter a blocked path, a failed script, an empty HTML shell, a redirect loop, or important facts that exist only inside an image.

You do not need a private crawler or perfect knowledge of every AI system to find the most important problems. A disciplined audit can show whether your key business pages are available, readable, and connected.

Start with a small, important page set

Do not begin by auditing every URL. Choose the pages an answer engine would need to recommend you: the home page, each primary service page, the main location page, contact page, about page, and two useful proof or process pages. Write down each canonical URL and the business facts it should communicate.

For every page, record whether it has:

• A clear service or business purpose in the title and opening copy.
• Plain-text details about location, service area, qualifications, and contact options.
• A stable URL that loads without a login, form submission, or special browser state.
• Links from another indexable page on your site.
• A canonical tag pointing to the intended version, if your platform uses canonicals.

This inventory gives you a test set. It also stops a technical cleanup from losing sight of the customer questions the pages must answer.

Step one: check robots and response behavior

Open your robots.txt file at the root of the domain. Look for broad disallow rules and rules that prevent access to the paths in your test set. Robots directives are not the only control an engine may use, and policies differ by crawler, so do not assume that changing one line guarantees access. Treat the file as a first diagnostic.

Use a command-line request or a developer tool to inspect each URL's response. You are looking for a successful response, a reasonable redirect chain, and the expected canonical host. A page that returns an error to a crawler is not fixed merely because a browser later displays a cached copy.

Also check for accidental controls outside robots.txt: password protection, IP restrictions, bot challenges, cookie walls, geo blocks, or a server rule that treats unknown user agents differently. Ask your developer or hosting provider to review logs if you see inconsistent behavior. The aim is not to let every automated request through; it is to ensure legitimate discovery and retrieval of public business information are not unintentionally blocked.

Step two: compare raw HTML with the rendered page

View the page source, not only the inspector's final DOM. Search the source for the exact service name, location, phone number, and one sentence that explains the offer. If those facts appear only after JavaScript runs, the page may still work for modern systems, but it has an avoidable dependency.

Use a text-only fetch or browser setting that disables scripts, then compare the result with the normal page. A robust page should retain its purpose, key text, headings, links, and contact route. Decorative components, booking widgets, and testimonials can be enhanced with JavaScript; the core explanation should not disappear when the script fails.

Watch for content hidden behind tabs, accordions, sliders, or an interaction that has no ordinary link. Those patterns can be useful for people, but put the answer in the initial document too when it matters. Make image-based text available as real text, and give meaningful images accurate alternative descriptions.

Step three: trace discovery and canonical signals

A crawler can read a page and still fail to discover it. Follow links from the home page and navigation to each test URL. Remove orphan pages, and use descriptive anchors rather than repeating "learn more." Check that important pages are not marked noindex accidentally and that the sitemap lists the preferred URLs when a sitemap is maintained.

Review duplicate versions: trailing slashes, tracking parameters, staging hosts, HTTP versions, and old service URLs. Choose one public URL, redirect obsolete versions where appropriate, and make internal links consistent. A canonical tag is a helpful hint, not a substitute for clean architecture and correct redirects.

Step four: test the business facts directly

For each page, copy the first several paragraphs into a plain-text note. Could a person answer the following without looking elsewhere?

• What does this business do?
• Who is the service for?
• Where is it available?
• What happens next?
• What qualifications or constraints matter?
• How can someone contact the business?

If the answer requires interpreting a slogan, a photo, or a vague menu label, rewrite it. This is an information design issue as much as a crawler issue. Keep facts near the topic they describe, use normal headings, and avoid hiding the only location reference in a footer.

Step five: monitor after the fix

Save before-and-after copies of the URLs you changed. Ask your developer to verify server status, response headers, and logs after deployment. Re-run the text comparison and check internal links. Then use consistent, unbranded questions in answer engines to see whether your business is understood more accurately; this is an observation, not proof that a particular crawler changed its behavior.

Recheck after redesigns, migrations, new security rules, or a change to the content platform. Keep the audit in a simple spreadsheet with URL, issue, owner, date found, date fixed, and verification note. A readable page is not a guarantee of a citation, but it removes one important source of uncertainty and gives every future content improvement a better foundation.