What is llms.txt — and why Google says you don't need it

People write about llms.txt like it's the new SEO: "drop the file and AI will start citing you." The catch is that Google, on its own page about AI features, says exactly the opposite.
Let's unpack what this file is, where it came from, and when it's genuinely worth making.
Why anyone wanted another file
The idea is simple and honest. The spec frames the problem like this:
Web pages are built for people. An HTML page wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise.
And the other half: "Context windows, while larger than they were, are still too small for most websites in their entirety."
So a model physically can't swallow your whole site. And half of what it does read is menus, banners and cookie buttons.
llms.txt proposes the blunt fix: put a short text file at your site root that says, in plain words, what lives here and where to go for detail. The author is Jeremy Howard; the spec was published on 3 September 2024. Today it turns exactly two.
How it's built inside
It's ordinary Markdown, not a new syntax. The spec fixes the order of sections:
- a level-one heading with the project name — the only required part;
- a blockquote with a short summary;
- as many plain paragraphs of detail as you like;
- link lists under level-two headings.
It looks like this:
# KODiQ
> An app that teaches you to build things with AI from zero.
## Docs
- [What is a prompt](https://kodiq.ai/article/what-is-a-prompt/): the base concept
- [How to deploy](https://kodiq.ai/article/what-is-deployment/): from code to an address
## Optional
- [Blog](https://kodiq.ai/feed/): full articles
A useful detail nobody mentions: the file doesn't have to sit at the root. The spec allows any path — it then covers the pages under that path. Handy when your docs live in their own section.
So who reads it
This is where it gets interesting.
The spec's own site claims that "thousands of sites publish an llms.txt file, documentation platforms generate one automatically, and Chrome's Lighthouse audits sites for one."
Google, in its guidance for site owners, says plainly:
You don't need to create new machine readable files, AI text files, or markup to appear in these features.
And on the same page: "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary."
This isn't a matter of taste. These are two different scenarios that everyone mixes up:
- Search and the search engine's AI answers. The file is not needed. The crawler arrives via ordinary links, reads ordinary HTML and decides for itself.
- An agent you handed the address to. Here it's needed and it works. You give the link to an AI agent, and it pulls a short index instead of crawling forty pages of navigation.
When to make one, and when not to
Worth it if you have something an AI reads because you pointed it there: documentation, an API, a product help centre, a library. The win is real — the agent gets clean text instead of markup soup.
Not worth it if the goal is "get into AI search" or "rank higher on Google." That's the job of SEO and a plain readable page, not an extra file at the root.
One honest warning: you'll maintain this file by hand. A dead llms.txt pointing at pages that no longer exist is worse than none at all.
There's a separate breakdown of the neighbouring file that everyone reads, always — robots.txt vs llms.txt.
The mistake that kills the whole idea
The most common thing people do with llms.txt: dump the entire site text into it. Whole articles, full section descriptions, a mile of prose — "let the model read everything at once."
Don't. Here's why. The spec was born from exactly the problem that "context windows are still too small for most websites in their entirety." Put the whole site in the file and you've recreated that problem, now in a single lump.
The format is designed as an index, not an archive. Look at the structure again: a name, a short summary, link lists with explanations. The model reads the index, understands the map, and follows a specific link for detail — precisely when that detail is needed.
Practical rule: if the file doesn't fit on a couple of screens, you're not writing an llms.txt, you're writing a second website.
Checking someone else's takes three seconds: append /llms.txt to the address of any service with documentation. You'll also see how people who've done this for a while lay it out.
Will llms.txt get me into ChatGPT's answers?
Not directly. It helps once a model has already opened your site: it hands over a short clean index instead of heavy HTML pages.
Is it an official standard?
No. It's an open proposal, not a standard from any consortium. Nobody is obliged to read it, and there's no penalty for not having one.
How is llms.txt different from a sitemap?
A sitemap is a list of addresses for search crawlers — machine XML. llms.txt is text meant to be read by a model: a name, a summary, and a note on why each link matters.
The AI Architect 2027 course: step by step from computer basics to your own app and first customers. 543 short lessons on your phone.





