Why AI Cannot Read Your Website

Being online is not the same as being readable

A buyer researching suppliers no longer starts with ten blue links. They describe the problem to an assistant, read the summary it produces and follow two or three of the sources it cites. If your company is absent from that summary, you are absent from the shortlist, no matter how well your site performs in a traditional search result. That is the practical reason to ask why AI cannot read your website: the question is not about vanity, it is about whether your pages are still reachable at the exact moment a decision is being formed.

The confusion usually comes from a false assumption. People check a page in their own browser, see a polished layout and conclude that the content exists. What exists in a browser is the end product of dozens of steps: a request that was accepted, a firewall that stayed quiet, scripts that ran, fonts and images that loaded. A language model receives something far more primitive, usually a single raw response. Whatever cannot be extracted from that response simply is not there.

Three different doors, three different failures

Assistants can reach your content through separate channels, and each one fails in its own way. Diagnosing the wrong channel is the most common waste of effort in this work.

• Training data. Text collected long before the conversation happens. You have almost no short-term control here, and chasing it is rarely a good use of budget.
• Retrieval through a search index. The assistant queries a search engine, receives a set of results and reads a few of them. Presence in that index is necessary but not sufficient; the page still has to survive being read.
• Live fetching. A user pastes a link, or the assistant follows one. Your server answers in real time. This is the most honest test you will ever get, because there is no cache, no index and no second chance.

These channels arrive under different identities. Server logs will show names such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Bingbot. Some collect content, some serve a search index, some act on behalf of a user in the moment. Treating all of them as a single category of unwanted traffic is a decision with consequences, and it is often made by accident rather than on purpose.

Failure one: the request never gets through

The most frequent reason AI cannot read a site has nothing to do with writing. The content is fine; nobody ever received it. The request was rejected at the door.

Four mechanisms account for nearly all of these cases. A robots.txt file, often written years ago and never revisited, may exclude specific clients. A bot protection or firewall rule, added during an attack and never relaxed, may serve a challenge page to every unfamiliar client; an assistant cannot solve a challenge, so all it receives is the text of the challenge itself. Rate limiting cuts a crawl in half when requests arrive faster than a threshold allows. And intermittent server errors mean the page can be perfectly healthy when you visit and unavailable in the minute a crawler arrived.

None of this requires speculation. Access logs answer it directly: search the last few months for those client names, then look at the status codes they received. A long run of 403 or 429 responses is not a mystery, it is a configuration decision showing its effects.

Why AI Cannot Read Your Website: Five Layers That Fail in Order

Suppose the door is now open. Four obstacles remain, and they occur in a fixed sequence: whether the text exists in the raw response, whether that text carries any extractable fact, whether the page can be attributed to an identifiable organisation, and whether it can be cited at a stable address. The rest of this article walks through each of them with a check you can run yourself.

Failure two: the text lives inside JavaScript

Many modern stacks send a nearly empty shell from the server and let client-side code assemble the page afterwards. Visitors never notice. Content-collecting clients notice immediately, because they are not obliged to behave like a full browser and commonly do not execute page scripts at all.

The test takes a minute. Fetch the page as raw markup and search that output for a sentence you know appears on screen. If the sentence is missing, that paragraph does not exist for any client that does not run scripts. The same applies to text hidden behind tabs, sections loaded on scroll, accordion panels that fetch their own content, and information that only appears inside an image. A specification table published as a screenshot is one more place AI cannot read a single figure.

Rebuilding the whole front end is rarely the answer. What matters is that the informative core of the page, the headings, the descriptions, the lists, the specifications and the answers to common questions, is present in server-rendered markup. Interactivity can stay exactly where it is.

Failure three: the page contains claims, not facts

Plenty of pages pass the first two tests and still never appear in an answer. The reason is usually that there is nothing on them worth quoting. A sentence such as "we deliver innovative solutions tailored to your needs" is grammatically fine and informationally empty.

When an assistant composes an answer it looks for statements it can lift and attribute: what a service includes and excludes, how long it takes, how the process runs, which cases it is unsuitable for, what drives the price up or down, which markets or company sizes are served, what certifications or standards apply. If a page carries none of these, a model may read it completely and still have nothing to say about you.

There is a quick diagnostic for this. Open your own service page and mentally delete your company name. If the remaining text could belong to any competitor, there is nothing distinctive to cite. Run the same test on the competitors that do get cited and the difference becomes uncomfortable but obvious. Among all the reasons why AI cannot read your website, this is the one that costs the least to fix and gets postponed the most.

Failure four: the organisation behind the page is unclear

To mention a company in an answer, a model needs a coherent picture of who that company is. If the legal name appears in three variations across the site, if the address differs between your pages and third-party listings, if what you actually sell is stated only inside a hero image, which AI cannot read, there is no entity to assemble.

The fixes here are unglamorous. State the company name, the field of activity and the contact details as real text on every page. Publish an about page that contains information rather than adjectives. Add structured data so that organisation, service and question-and-answer content is available in a form a parser can read without guessing. A plain-language summary file describing your site to assistants completes the picture, and we covered how to build one in our guide to llms.txt.

Companies often discover that this step alone changes results without a single word of new content being written. The material was always good enough; it simply could not be tied to anyone.

Failure five: the page is not citable

An assistant that wants to link to you needs an address that is stable and unambiguous. Several URLs serving the same content, addresses that change with every redesign, redirect chains that pass through three hops, or an interstitial that greets every visitor before the content appears, all reduce the chance that a page is offered as a source.

The checklist is short. One canonical address per piece of content. Redirects that reach their destination in a single step. Page titles that match what the page actually contains. Important information written into the page rather than locked inside a downloadable document. Anything published only as a PDF is one extra barrier away from being quoted.

An audit you can run this week

Every failure above is measurable, and none of the measurements require specialist tooling.

• Read your robots.txt line by line. Keep the exclusions you actually want and remove the ones you inherited.
• Search your access logs for assistant client names and record the status codes they received.
• Fetch key pages as raw markup and confirm that your actual sentences are present in the response.
• Review firewall and bot protection rules to see who receives a challenge and at which threshold rate limiting begins.
• Ask the assistants directly, using the questions your customers would ask, and record which sources they cite. We describe a repeatable method for this in our article on how to measure AI visibility.

The output of this audit is a written list, not an impression. You end up knowing which door is closed, which page AI cannot read and which fact was never published. We ran exactly this exercise against our own properties and published the uncomfortable parts in the report on asking AI about our own websites.

Fix them in the right order

Sequence matters, because each step depends on the one before it. Confirm access before writing anything; AI cannot read the best article behind a closed door. Once requests get through, verify that your text appears in the raw response. Once the text is visible, make sure it carries facts. Once the facts are there, complete the identity and structured data work. Leave the citability cleanup for last, since it is the cheapest and fastest part of the job.

A week of technical correction frequently produces more visibility than a quarter of content production. The reverse also holds: content published while the door is shut performs exactly like content that was never written.

This work is entirely doable in-house. If you would rather hand it over, the checks we run and the items we correct are listed on our AI visibility service page.

What to measure afterwards

Removing a barrier is not a result until something is measured. Three signals are worth watching. First, the logs: are assistant clients arriving now, and are they receiving successful responses? That is direct evidence that the door opened. Second, your standing in answers: ask a fixed set of questions at regular intervals and record whether your brand is named and whether your pages are cited. Third, the traffic those answers send, which can be tracked separately from ordinary search visits.

Measuring once tells you very little, because both the models and your site keep changing. If you would rather not run that cycle manually, our AI visibility tracking service records the questions, the answers and the cited sources on a schedule, so the output is a dated log rather than an opinion.

None of this is mysterious. It is ordinary fault-finding applied to a new channel: access, markup, substance, identity, citability. Work through those five and the question of why AI cannot read your website stops being rhetorical, because each answer is either an item you have already corrected or a specific item still waiting on your list. Looking good on a screen is no longer the whole job; a machine has to be able to extract the same meaning from the same page.