robots.txt vs llms.txt — which file do bots actually read

They get lined up together constantly: robots.txt is the old file for search engines, llms.txt the new one for AI. Get with the times, add the second one.
Here's the catch: these aren't the old and new versions of the same thing. Their jobs are opposite.
A fence and a signpost
robots.txt tells a bot where not to go. It's access control. Google describes it like this: "A robots.txt file tells search engine crawlers which URLs the crawler can access on your site," and adds that it's "used mainly to avoid overloading your site with requests."
llms.txt does the exact reverse: it tells a model where to go. It's navigation. A short Markdown text: what this project is, plus a list of links explaining why each one matters.
A fence and a signpost. You can have both, but they don't replace each other — and they don't compete.
Point by point
Who reads it. robots.txt — every search crawler, AI crawler and scraper. Fetching that file is the first thing any self-respecting bot does. llms.txt — whoever you handed the address to: an agent, a script, a docs-reading tool. Nothing crawls it by default.
How binding it is. This is the key difference. robots.txt is obeyed voluntarily, but every major player obeys it. Google is honest about the mechanism: "The instructions in robots.txt files cannot enforce crawler behavior to your site; it's up to the crawler to obey them." Nobody is obliged to honour llms.txt at all: it's an open proposal, not a standard.
What happens if the file is missing. Without robots.txt a crawler assumes everything is fair game and walks everywhere — including admin pages and heavy sections. Without llms.txt nothing happens: the model just reads ordinary HTML, as before.
Format. robots.txt uses strict directives: User-agent, Disallow, Allow, Sitemap. One typo and a rule silently stops working. llms.txt is free-form Markdown with exactly one required part: a level-one heading with the project name.
Where it lives. robots.txt — at the site root only, no alternatives. llms.txt — at the root or at any path, in which case it covers the pages beneath that path.
Does it do anything for search. robots.txt affects crawling and, through it, indexing. llms.txt affects search not at all: Google's guidance for site owners says plainly, "You don't need to create new machine readable files, AI text files, or markup to appear in these features."
Who should use what — the straight answer
Always ship robots.txt. Even for a five-page site. A minimal file that allows everything and points at your sitemap is two lines and settles the question for years. It's also the only place where you can ask AI crawlers not to take your content.
Ship llms.txt if you have documentation, an API, a product help centre or a library — something an agent reads because you pointed it there. Then it gets a clean index instead of forty pages of navigation.
Don't ship llms.txt if the goal is "rank higher" or "get into AI answers." You'll spend an evening and end up with a file you now maintain by hand. A dead llms.txt full of links to deleted pages is worse than none.
There's a full breakdown of the second file in its own piece: what is llms.txt.
What neither of them does
Neither file hides a page. For robots.txt the docs say it outright: "it is not a mechanism for keeping a web page out of Google." And more than that — "a page that's disallowed in robots.txt can still be indexed if linked to from other sites."
If a page genuinely must be hidden, there are two tools: a password, or a noindex meta tag. The second one, incidentally, only works if the crawler is allowed to read the page — block it in robots.txt and you've stopped the bot from ever seeing your own instruction.
And neither will stop a bad-faith scraper. Both files are a request, not a lock.
The minimal robots.txt you can ship right now
Two lines settle it for an ordinary site:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
The first pair lets every crawler in. The sitemap line is a hint about where to find your list of pages; both Google and other engines look for it.
If you need to close off an admin section, add Disallow: with the specific path — and remember it's a request not to visit, not a ban on showing.
Does a small site need robots.txt?
Yes, at least a minimal one. It cuts pointless crawler load and points at your sitemap — two lines of text.
Can I use both files at once?
You can, and often should: they don't conflict and live independently. Just don't expect the second to amplify the first.
Does llms.txt replace structured data?
No, they're different layers. Markup describes facts inside one specific page; llms.txt describes the shape of the whole site for a model to read.
The AI Architect 2027 course: step by step from computer basics to your own app and first customers. 543 short lessons on your phone.





