How to create your robots.txt
Start from a blank file or a preset for WordPress or an online store, add any folders you don't want crawled, choose which AI crawlers to block, and add your sitemap address. The table under the file shows what each crawler may visit, so you can check the result before you upload it.
Save the file as robots.txt in the top folder of your site, so it opens at yourwebsite.com/robots.txt. On Shopify, edit the robots.txt.liquid template instead; on Wix and Squarespace, use the SEO settings.
What to block, and what not to
Block pages with no search value that crawlers waste time on: admin areas, carts, checkout, internal search results and filter combinations. Don't block CSS, JavaScript or image folders, because Google needs them to see your pages properly.
Don't use robots.txt to hide private pages. The file is public, and blocked pages can still appear in results if they're linked from elsewhere. Use passwords or noindex instead.
Blocking AI crawlers
AI training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot) collect pages to train models. Blocking them doesn't change your Google or Bing rankings. AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch pages to cite them in answers with a link, so most sites keep them allowed.