Every site's robots.txt should be small and specific. These examples cover the cases most sites need. Replace example.com with your domain, and always test a new file before you publish it.
1. Allow everything
The simplest valid robots.txt. Every crawler can visit every page.
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlAn empty Disallow: means nothing is blocked.
2. Block one folder
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml3. Block a folder but allow one page inside it
User-agent: *
Disallow: /members/
Allow: /members/join/The longer, more specific Allow rule wins for /members/join/.
4. WordPress
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlKeep admin-ajax.php open: some themes and plugins use it on the front end. Don't block /wp-content/ or /wp-includes/, which hold the CSS, JavaScript and images Google needs.
5. Online shop
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /*?sort=
Disallow: /*?filter=
Sitemap: https://example.com/sitemap.xmlThe wildcard rules stop crawlers wasting time on endless sorting and filter combinations. Make sure your category pages themselves stay crawlable. (Shopify generates its own robots.txt with similar rules; you can edit it with a robots.txt.liquid template.)
6. Block internal search results
User-agent: *
Disallow: /search
Disallow: /*?s=7. Block a file type
User-agent: *
Disallow: /*.pdf$$ means "ends with", so this blocks addresses ending in .pdf.
Build your own
Use the generator on this page to tick the sections you want to block, choose which AI crawlers to allow, add your sitemap and copy a ready file. Then keep reading for the special cases.