What is robots.txt, and how does it work?

Every well-behaved crawler, from Googlebot to AI bots, checks a site's robots.txt file before crawling it. The file lives at the root of the domain, such as https://example.com/robots.txt, and contains rules about which paths each crawler may visit.

A simple robots.txt

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

This tells every crawler (*) not to crawl the /admin/ and /cart/ sections, and points them to the sitemap.

The main directives

Google also supports two wildcards: * matches any sequence of characters, and $ marks the end of a URL. For example, Disallow: /*.pdf$ blocks every PDF.

How Google reads the rules

• Each crawler follows the most specific group that names it. If there's a User-agent: Googlebot group, Googlebot ignores the * group entirely.
• Within a group, the most specific rule wins: the one with the longest matching path.
• If an Allow and a Disallow match equally, Allow wins.
• Paths are case-sensitive.
• If robots.txt returns a 404, Google assumes it can crawl everything. If it returns a server error for a long time, Google may stop crawling the site.

What robots.txt can't do

This is where most mistakes come from.

1. It doesn't remove pages from Google. A blocked page can still be indexed if other sites link to it; it just appears without a description. To keep a page out of results, use noindex and leave it crawlable.
2. It isn't security. The file is public, and bad bots ignore it. Never list secret paths there; protect them with a login.
3. Google ignores crawl-delay and noindex lines. Bing respects crawl-delay; Google doesn't.
4. It only applies to its own host. shop.example.com needs its own robots.txt.

Common robots.txt mistakes

• Disallow: / left over from a staging site, which blocks everything.
• Blocking CSS and JavaScript, which stops Google rendering the page properly.
• Blocking pages you want removed instead of noindexing them.
• Relative sitemap URLs: the Sitemap line must be a full address.
• Typos like Dissallow or User agent, which crawlers skip.

Check your robots.txt

Enter your site in the checker on this page. It fetches your robots.txt, validates every line, flags rules that block important pages or resources, and checks that your sitemaps load. To test whether a specific address is blocked for a specific crawler, use the robots.txt tester. To write a new file from scratch, see robots.txt examples.