llms.txt Explained: The Complete Guide
Published
· Updated
llms.txt is a Markdown file at the root of your domain that hands AI systems a curated map of your key pages. It is a community convention, not a standard. Google says it does not use the file, and 97% of published files receive no requests at all. Ship one for developer documentation. Skip it for a marketing site.
What llms.txt actually is
The premise is reasonable. A language model has a finite context window, and a modern web page is mostly not content. Navigation, cookie banners, related-post modules, and footer links all have to be parsed and discarded before the model reaches the part that answers the question. Converting messy HTML to clean text is, in the spec's own words, difficult and imprecise.
llms.txt proposes a shortcut. Publish one Markdown file at a predictable location that says what the site is, what it covers, and which pages matter. A model that wants your content can read that file first and go straight to the right pages.
That is the whole idea. It is a curated map, not a permission system and not a ranking signal.
The file format, exactly
The spec is deliberately thin. Only the H1 is required. Everything else is optional but recommended.
Element | Required | What it does |
|---|---|---|
H1 | Yes | The name of the site or project. Just the name, not a tagline |
Blockquote | No | A short summary of what the site is and who it serves. This is the passage most likely to be extracted if the file is read |
Free-form Markdown | No | Any prose that helps interpret the file. Any block type except headings |
H2 sections | No | Group links under headings such as Docs, Guides, or API |
Link list items | No | A Markdown link, then optionally a colon and a note about the page |
## Optional section | No | A reserved heading marking material an agent can skip when its context budget is tight |
A working file looks like this:
The spec also proposes serving clean Markdown mirrors of individual pages at the same URL with .md appended. That part is even less widely adopted than the file itself.
There is a companion convention, llms-full.txt, which inlines the full text of those pages into a single document so an agent can take everything in one fetch. Most sites do not need it. Documentation sites are the exception.
llms.txt vs robots.txt vs sitemap.xml
These three get confused constantly, usually because they share a location and a file extension. They do unrelated jobs.
robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
Question it answers | What may you access? | What URLs exist? | What matters, and what is this site? |
Audience | Crawlers | Search indexers | LLMs at inference time |
Status | Formal standard (RFC 9309) | Formal standard | Community convention |
Format | Plain text directives | XML | Markdown |
Honoured in practice | Widely, by well-behaved bots | Yes | Rarely |
Coverage | Path rules | Every indexable URL | A curated subset |
The practical point: llms.txt cannot block anything. If your goal is to control which AI crawlers reach your content, robots.txt is the file with real support, and the CDN layer is where enforcement actually happens. Our robots.txt generator covers the AI crawler directives.
Does anything actually read it?
This is the part most guides skip, and it is the only part that should decide whether you spend time on this.
Google has said no, on the record. Gary Illyes confirmed in July 2025 that Google does not support llms.txt and is not planning to. John Mueller compared it to the keywords meta tag, which is the canonical example of a self-declared signal that search engines stopped trusting fifteen years ago. Google's official generative AI guide, published on 15 May 2026, lists llms.txt in its mythbusting section as something you do not need for AI Overviews or AI Mode. Google may crawl the file like any other page, but says it is not treated in any special way.
No other major provider has committed either. OpenAI, Anthropic, Meta, Mistral, and Perplexity have not publicly stated that their production systems read or act on the file. Anthropic publishes its own llms.txt without claiming its crawlers consume the standard.
The server logs are the blunt part. Ahrefs studied 137,210 domains with measurable traffic in May 2026 and found that roughly 97% of llms.txt files received zero requests. The same study noted that AI bots never requested the file on domains where it did not exist, meaning they are not probing for it. A separate monitoring analysis of more than 500 million AI bot events over a 90-day window found only a few hundred requests targeting /llms.txt directly. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended overwhelmingly fetch HTML instead.
Correlation studies find nothing. An SE Ranking analysis of roughly 300,000 domains found no relationship between having the file and being cited in AI answers. One analysis reported that removing llms.txt as a variable actually improved citation prediction accuracy, which is a polite way of saying it adds noise.
Most of the traffic these files do receive comes from SEO audit tools checking whether the file exists.
Where it genuinely earns its place
There is one real use case, and it is not marketing.
AI coding assistants such as Cursor, GitHub Copilot, and Claude Code fetch documentation live while a developer works. They are pointed at your docs directly, not discovering them through a search index. In that flow, a curated entry point saves tokens and materially improves the answer the developer gets. Vercel is the example most often cited: their file carries contextual descriptions that let an agent decide which API pages to fetch before fetching anything.
So the honest decision rule looks like this:
Your site | Ship llms.txt? | Why |
|---|---|---|
Developer documentation, API reference, SDK | Yes, plus llms-full.txt | Your users' tools consume it directly |
SaaS product with a docs section | Yes, scoped to docs | Same reason, narrower scope |
Marketing site, blog, ecommerce | Optional, low priority | Nothing in the retrieval path reads it |
Site with unresolved crawling or rendering faults | No, fix those first | The file cannot compensate for pages an engine cannot read |
If you fall in the last row, the work that does move AI visibility is covered in our LLM SEO guide.
How to create and check one
You can write the file by hand in ten minutes. If you would rather generate it from your existing sitemap with real page titles and descriptions, our llms.txt generator does that, and the llms.txt validator checks an existing file against the spec for structure, link format, and broken links.
Plugins and platforms have started shipping support too. Yoast offers one-click generation, and Webflow allows you to upload the file to your root.
Whatever you use, three things matter: serve it at the exact path /llms.txt, serve it as text/plain or text/markdown rather than HTML, and keep the links live.
This is one piece of a broader AI-visibility toolkit. Browse all our posts for the rest.
Common mistakes
Dumping the sitemap into it. The file is a curated subset. If it lists every URL, it has no editorial value and you have just made a worse sitemap.
Filling the blockquote with marketing copy. The summary should state what the site is and what it covers. Positioning language wastes the one passage most likely to be read.
Letting it rot. A file listing pages you deleted eight months ago is worse than no file.
Treating it as access control. It blocks nothing. That is robots.txt.
Expecting it to lift rankings or citations. No evidence supports this, and Google has explicitly said it does nothing for its AI surfaces.
Important notes
This article is for educational purposes and reflects the position as of September 2026. This area moves quickly, and provider support could change.
Crawler and adoption figures cited here come from published third-party studies, including Ahrefs, SE Ranking, and independent bot-log monitoring. Methodologies differ, and none of them observes the internals of any AI provider's retrieval pipeline.
Statements attributed to Google are from public Search Central documentation and named Google staff. They describe Google's own AI surfaces and do not speak for other providers.
Read more from the team: browse all our posts.
Frequently asked questions
What is llms.txt?
A Markdown file at the root of a domain that gives AI systems a curated map of a site's most important pages, with a short summary of what the site is. Jeremy Howard of Answer.AI proposed it on 3 September 2024.
Does Google use llms.txt?
No. Gary Illyes confirmed in July 2025 that Google does not support it and is not planning to, and Google's May 2026 generative AI guide lists it among tactics that are not needed. Google may crawl the file like any other page, but does not treat it specially.
Do ChatGPT, Claude, or Perplexity read it?
None of their operators has publicly confirmed that production systems read or act on the file. Server log studies show their crawlers overwhelmingly fetch HTML instead, and rarely request /llms.txt at all.
Will llms.txt improve my AI citations?
There is no evidence that it will. A study of roughly 300,000 domains found no correlation between having the file and being cited in AI answers.
Is llms.txt the same as robots.txt?
No, and they are close to opposites. robots.txt is a formal standard that governs what a crawler may access. llms.txt is a convention that recommends what is worth reading. Keep both, for different reasons.
What is llms-full.txt?
A companion file that inlines the full text of your key pages into one Markdown document, so an agent can take everything in a single fetch. It is useful for documentation and unnecessary for most other sites.
Should I remove the file if it does nothing?
There is no need. It is harmless to serve, costs nothing to keep, and would be quick to benefit from if a provider adopts it later. The cost was never the file, it was treating it as a strategy.
How big should it be?
Small enough to be useful. For most sites, one screen of links across three or four sections. The point is curation, so a long file defeats the purpose.
Does it need to be valid Markdown?
Yes. The whole file is meant to be valid CommonMark. Broken structure makes it unparseable for the handful of tools that do read it.