What Should You Include in a robots.txt File?

Introduction

If you want to control how search engines and bots move through your site, you’ve got to know a little technical SEO. Robots.txt might look simple—just a tiny text file—but honestly, it’s your main lever for steering crawlers. It tells them which parts of your site to skip, and what they’re welcome to index. Doesn’t seem like much, but it holds a surprising amount of power.

Whether you’re dealing with search bots, fending off AI scrapers, or working with crazy URLs that switch up with every language, understanding robots.txt puts you in charge. You get to call the shots on what bots see and what gets left alone.

What Is robots.txt and Why Does It Matter?

Robots.txt is just a text file you toss into your site’s top directory. It’s filled with rules for automated crawlers—think of it as “authorized personnel only” signs for bots.

When you go to:

`https://www.example.com/robots.txt`

A bot drops by, reads the rules, and—if it’s behaved—follows what you wrote. Set things up right, and you can:

- Stop bots chewing through private or pointless pages
- Save your server from useless traffic
- Keep robots clear of your admin or duplicate content
- Point them to your sitemap
- Tell AI bots about your preferences (assuming they care)

A little tip: robots.txt blocks crawling, not indexing. If another site links to a blocked page, a search engine might still list it—they’ll just won’t show its contents.

Robots Exclusion Protocol (RFC 9309): The Basics

RFC 9309 spells out the official expectations for robots.txt. It describes how bots should read the file, match rules to their user-agent, and know what’s allowed or restricted.

A basic robots.txt file looks like this:

```

User-agent: *
Disallow: /admin/
Disallow: /temporary/
Allow: /public/
Sitemap: https://www.example.com/sitemap.xml
```

That means every bot (matched by the *) should stay out of `/admin/` and `/temporary/`, but they’re fine to visit `/public/`.

You can get more granular too, writing rules for particular bots. But just remember—not every bot actually plays nice. Always check how your setup works in practice.

How Does a robots.txt Parser Read Your File?

When a bot arrives, it scans your robots.txt and follows the rules matched to its user-agent.

There are a few core commands:

- `User-agent`: pick the bot you’re talking to
- `Disallow`: tell it where NOT to go
- `Allow`: let it access specific spots, even inside blocked folders
- `Sitemap`: show bots where your sitemap lives

So if you write:

```
User-agent: *
Disallow: /members/
Allow: /members/help/
```

This blocks bots from most of `/members/`, except for `/members/help/`. Here’s the thing: not all crawlers care what you write, and a syntax mistake means your rules get ignored. Stick to proper commands or bots will do as they please.

What About Comments and ASCII Art?

You can add comments using `#`. They’re handy for quick notes:

```
Block internal search results
User-agent: *
Disallow: /search/
```

Feel free to add ASCII art if you’re feeling fancy, but simple is best. Comments don’t affect bots—they just help people figure out why you set certain rules.

Using Emojis and International Characters in robots.txt

Some websites use non-standard characters or emojis in their URLs—accents, non-Latin letters, you name it. That’s fine for your content, but robots.txt wants precision.

Most URLs use ASCII, so if you’ve got something like `café`, you need to percent-encode it as `/caf%C3%A9/` in your rules. If your CMS spits out weird URLs, always grab the encoded version for robots.txt.

If you skip this, your rule won’t match. Always:

1. Check your actual site paths,
2. Get the encoded version,
3. Drop it into robots.txt,
4. Test with real bots or a robots.txt checker,
5. Watch for strange bot behavior—some don’t follow the rules.

Guessing isn’t good enough—double-check every time.

Controlling AI Crawlers With robots.txt

AI bots love crawling sites for training data. Sometimes you’re fine with it, sometimes you want to shut them out.

If an AI crawler listens to robots.txt, blocking it is easy:

```
User-agent: ExampleBot
Disallow: /
```

Just swap “ExampleBot” for their real user-agent. This blocks the bot entirely. To limit just a section:

```
User-agent: ExampleBot
Disallow: /restricted-content/
Allow: /restricted-content/public/
```

But every AI bot is different. Check their docs and see how they actually behave.

Will Blocking an AI Bot Protect Your Content?

Not always. robots.txt only stops bots from crawling—it can’t claw back anything they already scraped. If you care about your content, you’ll want stronger technical or legal defenses. Some companies even run a bunch of bots with unrelated names, so blocking one doesn’t guarantee privacy.

Common robots.txt Mistakes That Hurt SEO

Mistakes happen. Here are a few that can wreck your SEO:

1. Blocking the Whole Site

A `Disallow: /` stops every bot from crawling anything. Unless you really want your site invisible, don’t do this.

2. Path Matching Errors

Miss a capital letter, forget a trailing slash, or botch percent encoding? Your rule won’t work.

3. Thinking Blocked Pages Stay Private

Everyone can see your robots.txt, including bad bots and nosy people. Don’t trust it with actual secrets.

4. Blocking Important Resources

Lock down CSS, JavaScript, or images, and search engines may have trouble rendering your site. That hurts your rankings.

5. Messing Up the Syntax

Bots ignore what they can’t understand. Stick to the correct commands or they’ll just skip over your rules.

6. Using robots.txt for Noindex

Blocking a page doesn’t mean it disappears from search results. If you want a page out, use the `noindex` tag.

Technical SEO Best Practices

Here’s how to keep robots.txt working in your favor:

- Make it simple, readable, and organized
- Block only what you really want to hide
- Write bot-specific rules only when needed
- Double-check percent-encoded paths
- Always include your sitemap URL
- Test your rules before publishing
- Monitor bot activity in webmaster tools or server logs
- Use proper security for private data—don’t rely on robots.txt

Conclusion

Robots.txt seems basic, but it’s a must-have for technical SEO. Use the right commands, shape your rules for international URLs, and you’ll manage bot traffic reliably.

But remember, it’s just a suggestion—well-behaved bots listen, others might not. It can’t scrub out anything that’s already been scraped or guarantee total privacy.

Follow the standards, test every change, and dig into bot documentation when you need to. This way, you guide good bots, block the ones you don’t want, and sidestep the usual SEO headaches.