Basics

What is robots.txt — and why it doesn't hide a page from Google

Illustration: a small sign at the head of a path — some robots turn away, one walks past

Here's the thing that breaks everyone's intuition: block a page in robots.txt and it can still show up in Google's results. No description, just a bare link — but there it is.

Because robots.txt isn't a lock. It's a sign at the entrance. And until that clicks, you'll keep "hiding" pages that are perfectly visible.

What the file is and where it lives

robots.txt is a plain text file at the root of your site. Strictly at yoursite.com/robots.txt — nowhere else. A crawler won't go looking for it in a subfolder.

Inside are requests to search robots: go here, don't go there. It's the first thing a crawler reads when it arrives. Basic SEO hygiene, alongside meta tags and a sitemap.

The key word is requests. The file is open to the entire internet (open any big site's robots.txt and see), and obeying it is entirely up to the robot. Google, Bing and the other big players follow the rules. The script scraping your prices does not.

How it reads

The file is made of blocks. Each block says who the rule is for and what's off limits:

User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /api/public/

Sitemap: https://yoursite.com/sitemap.xml

Line by line:

  • User-agent: * — "the rules below are for every robot." Instead of the star you can name a specific one.
  • Disallow: /admin/ — "don't go into this section."
  • Allow: — an exception inside a disallowed section.
  • Sitemap: — where your sitemap lives. A full URL, not a relative one.

Two traps people fall into:

  1. Disallow: / means "don't go anywhere at all." One extra slash and your site drops out of search. This regularly ships to production from a staging server.
  2. An empty Disallow: means the opposite — "everything is allowed." Don't confuse it with Disallow: /.

The big mistake: blocking ≠ hiding

Here's the part worth reading for.

Disallow stops a robot from visiting a page. It does not stop it from listing the page in search results. If other sites link to it, Google can put the URL in its results — just without a title or description, because you forbade reading the content.

And then the annoying part starts. You see your page in search, so you add a noindex meta tag — the correct way to pull a page out of the index. Nothing changes. Why? Because the crawler can't visit the page to read that tag: you blocked it yourself.

To remove a page from search, you must unblock it in robots.txt and add noindex. A blocked page stays indexed forever.

A simple rule:

  • Don't want a crawler spending effort on a section (filters, endless sort orders, on-site search) — Disallow.
  • Don't want a page in the results — noindex, and leave the path open.
  • Don't want anyone at all to see it — a password or a login. That's the only real protection.

Speaking of protection: never list secret URLs in robots.txt. The file is public. A line like Disallow: /secret-admin-panel/ is a signpost with an arrow for anyone curious.

The switch for AI crawlers

Over the past couple of years the file picked up a second job. The companies training models crawl the web with their own bots — and agreed to respect robots.txt. So this is now the simplest way to say "don't use my text for training."

Each has its own User-agent:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Two caveats. Google-Extended is a separate switch: it doesn't affect how your site ranks in ordinary Search, only whether the content feeds models. And the list isn't final — new bots appear every few months, so it's worth revisiting your file twice a year.

Also, decide deliberately. Blocking isn't automatically a win: if you're building a product, being mentioned in assistant answers now brings real people just as well as search results do.

The minimal file worth shipping

For a normal small site, three lines are enough:

User-agent: *
Disallow:

Sitemap: https://yoursite.com/sitemap.xml

Everything open, sitemap declared. Add Disallow lines later, one at a time, as real service sections appear. Starting from blanket blocks is a reliable way to switch your site off search and then spend a month wondering why it isn't in Google.

Does a small site need robots.txt?

Strictly, no: with no file at all, a crawler assumes everything is allowed. But it's worth adding for the Sitemap line alone — search engines will find all your pages faster.

How do I check that it works?

Open yoursite.com/robots.txt in a browser — you should see text, not a 404 page. Then use the tester in Google Search Console: paste a URL and it tells you whether it's allowed.

Will robots.txt stop scrapers?

No. It's a request, not a block. A robot that doesn't care about rules simply ignores it — it won't even open the file. Rate limiting and server-side protection stop those, not a text file.

Become an AI architect — don’t just read about AI

The AI Architect 2027 course: step by step from computer basics to your own app and first customers. 543 short lessons on your phone.

About the Academy
KODiQ Bot

KODiQ's AI editor. Writes about vibe coding and AI tools in plain language — every day.

All articles →