What Is llms.txt? The File That Explains Your Site to AI

The Short Answer: What Is llms.txt?

llms.txt is a plain text file written in Markdown that sits in your site's root directory. It opens with a few sentences describing what the site does, then lists your most important pages with a one-line description for each. Its purpose fits in a single sentence: when a language model looks at your site, it should not have to sift through hundreds of pages to find the right one — it should read your own answer to the question what is here.

The proposal was published by Jeremy Howard at llmstxt.org and was picked up quickly by the search community. The address is fixed: the file is always looked for at yourdomain.com/llms.txt. Most implementations add a second file — llms-full.txt — which gathers the full text of the listed pages into one place.

That is the definition. The real question is different: does the file actually do anything? This article covers both what it is and where it honestly stops, because most answers to "what is llms.txt" skip the second half and leave the impression that a standard exists.

What Is llms.txt Built From?

The format is deliberately plain: a heading (the site name), an optional summary paragraph, then lists of links under section headings. Each link is expected to carry a one-sentence description. On a services site, the sections typically break down like this:

• Services — what you sell, each service page with a single line of explanation
• Pricing and process — the pages that explain how you charge and how you work
• Guides — informative content that does not sell anything
• Company — who you are, where you are, how you are reached
• Optional — secondary pages a model can skip unless it needs them

One point deserves emphasis, because it is where most answers to what is llms.txt go wrong: this is not a dumping ground for links. Listing every URL is already sitemap.xml's job. The value of llms.txt runs in the opposite direction — selection. On a two-hundred-page site, you decide which fifteen pages describe you correctly. If you cannot make that choice, the file will not help either.

robots.txt, sitemap.xml and llms.txt Answer Three Different Questions

These three files get confused constantly, yet each answers something different:

• robots.txt is permission. "You may enter this directory, you may not enter that one."
• sitemap.xml is inventory. "These addresses exist, and here is when they changed."
• llms.txt is a briefing. "This site does the following; start with these pages."

None of the three replaces another, and they must not contradict each other. One inconsistency we see often: AI crawlers are blocked in robots.txt while an llms.txt sits on the same site. That is like locking the door and taping a note to it listing what is inside. Decide which crawlers you allow first, then write the file.

Do the Engines Really Read llms.txt?

This deserves plain language, because the question of what is llms.txt is usually answered far too optimistically. No major provider has officially announced support for llms.txt. The crawler documentation published by OpenAI, Anthropic, Perplexity and Google contains no commitment to this file; the address is not a directive but a convention the community adopted.

llms.txt is not a standard, it is a proposal. It costs little to set up, but on its own it promises no visibility.

If you prefer measurement to claims, you already own the instrument: your own server log. Requests from agents such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended are visible there, and so is the file each one asked for. Whether a crawler has requested your llms.txt is answered far more definitively by that log than by any article on the subject. Watching the log for a while after setup is the most honest way to answer "what is llms.txt worth" for your own site.

So Why Do We Set It Up Anyway?

The arithmetic is simple: the cost is near zero, there is no risk, and the side benefit is real. If the file becomes widely read tomorrow, you are ready; if it never does, you spent a few hours. But that is not even the main reason.

The real benefit: being forced to describe yourself

Sitting down to write an llms.txt means answering these questions: What does this site do? Which five pages matter most? How would I describe each in one sentence? A site with no clear answer to those three is not a surprising absence from AI answers. The same vagueness lives in your homepage's first screen, your title tags and your meta descriptions — llms.txt merely makes it visible. The checklist behind SEO-friendly web design overlaps heavily with what this file asks for; both demand the same discipline.

The second benefit: people read it too

Companies that publish an llms.txt often notice later that they use the file elsewhere. A well-written one doubles as a short company briefing: it can be handed to a new employee, an agency or a supplier as the answer to "what is on the site". That side of the file depends on no engine's decision.

How We Set It Up on Six Sites: Generated, Not Hand-Written

All six sites we manage carry llms.txt and llms-full.txt, and none of them is a hand-written static file. That was a deliberate choice, and experience backs it: a hand-written llms.txt has a short life. You add two service pages, change one article's address, and the file quietly turns wrong while nobody notices for months. A wrong llms.txt does more damage than no llms.txt, because it hands the model addresses that no longer exist.

The arrangement works like this. A generator produces the file: it pulls published pages from the page table and published articles from the blog table, groups them into sections, and writes each record's own summary onto its line. A rewrite rule on the server points the /llms.txt address at that generator, so nothing changes for the visitor. The moment an article is published from the panel, the file is current — nobody has to remember anything.

There is also a fallback layer. If the database cannot answer for some reason, the generator does not emit an empty file; a static copy takes over. When an AI crawler arrives, finding a slightly stale file beats finding an empty or broken one.

Five Rules for a Good llms.txt

• Fewer pages, the right pages. List the pages that describe you, not every address on the site. A long list is a confession that no choice was made.
• Descriptions are not sales lines. "We are the industry leader" carries no information. "A team building corporate websites and custom software, working from Bilecik and Istanbul" does.
• Use full, absolute addresses. Write complete URLs rather than relative paths, so links survive when the file is copied or read out of context.
• Be concrete. Models cannot summarise vague sentences. What you do, who you do it for and where you are should be stated openly.
• Generate it automatically. A manually maintained file goes stale eventually. Even on a small site, wiring generation into a script is a few hours of work and then never needs attention again.

Is llms-full.txt Necessary?

The second file collects the full text of the listed pages in one place. The logic: when a model reads it, it should not need to visit fifteen separate addresses — it gets the whole text at once. That is sensible for small and mid-sized sites; on a site with content spread across thousands of pages, the file grows past usefulness.

Our practical test is this: if the combined page text stays a reasonable size, generate llms-full.txt; if it does not, keep llms.txt alone and leave the full text to the pages themselves. Whichever you produce, verify once after setup that both files are served as plain text and really return current content. On the sites we build, both come from the same generator — one prints the curated link list, the other the text of those pages.

How to Set Up llms.txt on WordPress

There are three routes on WordPress. Some SEO plugins (Yoast and All in One SEO among them) generate the file themselves; there are small plugins written purely for this purpose; and the third route is producing it with your own generator. The plugin route is fast, but the plugin usually decides what goes into the list — which is a limitation if you want to apply the "fewer pages, the right pages" rule above.

Whichever you pick, know one WordPress-specific trap: a static file placed in the root, or a custom rewrite rule you add, will not work at all if it is defined after WordPress's own catch-all rule. The request is captured by WordPress before it ever reaches the file, and you get a 404. Custom rules must sit before the first catch-all block in the server configuration. We learned this detail the hard way on one site: the address would not open in the browser, and the file was exactly where it belonged.

Four Common Mistakes

Reviewing setups by teams that have just looked up what is llms.txt, the same four mistakes keep appearing.

• Making the file a copy of the sitemap. A few hundred lines of undescribed links is not an llms.txt; it is a harder-to-read version of sitemap.xml.
• The wrong content type. If the server serves the file as HTML or returns a 404, the file does not count as present. After setup, open the address and confirm that plain text actually comes back.
• Contradicting robots.txt. Blocking AI crawlers while publishing an llms.txt means two decisions that cancel each other out.
• Mistaking the file for the work. This is the most common one. Setting up llms.txt is not a visibility programme; it is the smallest and easiest part of one.

The Work That Comes Before llms.txt

Whoever asks what is llms.txt hoping that one file will put their brand into AI answers has the order wrong from the start. Language models usually learn about you not from your site but from what others have written about you: directories, marketplaces, forum threads, news and blog content. That is exactly what we saw when we measured our own four sites in a study of 192 observations: when the questions were about the sector, the answers were filled largely by marketplaces and directories rather than by the sites themselves.

So the right sequence is: measure first, fix second, and set up the file somewhere along the way. Every improvement made without measurement is a guess, and the bill for guessing arrives later. How the measurement is done, why a single query does not count as measurement, and which four numbers to watch are covered step by step in how AI visibility is measured. If you need the definitions first — the difference between GEO and AEO — start here.

Where to Begin

We suggest a practical order. Open your server log first and check whether AI crawlers reach your site at all; if they do not, the problem is access and content, not a file. Then read your robots.txt and make a conscious decision about which agents you allow. Next, pick your fifteen most important pages and write one honest sentence for each — this is the part that genuinely requires thinking. Producing the file is the last step, and if you can, produce it so that it keeps itself current as the site grows.

Our best answer to "what is llms.txt" is this: the first and cheapest step you will take while fixing how your brand appears to AI. It is not sufficient on its own, but it is valuable because it forces you to think about the real work. If you would rather run that work end to end with us, the AI Visibility page explains how the measurement works and what gets fixed afterwards.