How to Block AI Crawlers With Robots.txt

Most AI companies now run several crawlers with different jobs. Some collect pages to train models. Others fetch pages to answer a question in AI search, often with a link back to you. Blocking them all is easy, but it may cost you visibility in AI answers. The better approach is to decide bot by bot.

See which AI crawlers can reach your site

Enter your site in the checker on this page. It reads your robots.txt and shows, for each major AI crawler, whether it's allowed, blocked, or not mentioned at all.

The main AI crawlers

Crawler	Company	What it does
GPTBot	OpenAI	Collects content for model training
OAI-SearchBot	OpenAI	Finds pages to show and link in ChatGPT search
ChatGPT-User	OpenAI	Fetches a page when a user asks ChatGPT to
ClaudeBot	Anthropic	Collects content for model training
Claude-SearchBot	Anthropic	Indexes pages for Claude's search results
Claude-User	Anthropic	Fetches a page when a user asks Claude to
PerplexityBot	Perplexity	Indexes pages for Perplexity answers
Google-Extended	Google	A control token for Gemini training and grounding, not a separate crawler
Applebot-Extended	Apple	A control token for Apple's AI training
CCBot	Common Crawl	Builds an open web archive widely used for training
Bytespider	ByteDance	Collects content for ByteDance's AI models
meta-externalagent	Meta	Collects content for Meta's AI models

Bot names and roles change, so check each company's documentation before relying on a rule.

Training bots vs search bots

This distinction matters most:

Blocking a training bot keeps your content out of future model training. It doesn't usually affect whether you're cited in that company's AI search.
Blocking a search bot can remove you from that company's AI answers, which increasingly send referral traffic.

A common middle ground: block training, allow search.

Block AI training, allow AI search

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: *
Disallow:

Block all AI crawlers

Add a Disallow: / group for every bot in the table above. Keep your normal User-agent: * group for search engines.

What about Google's AI Overviews?

Google-Extended controls whether your content is used for Gemini models and grounding. It does not remove you from Google Search, and Google's AI Overviews in Search are governed by normal Googlebot crawling. The only way to keep a page out of AI Overviews is a nosnippet or noindex directive, which also affects your normal results. Blocking Googlebot would remove you from Google entirely.

Limits of robots.txt

Robots.txt is a request. Reputable companies follow it; some bots don't.
It only affects future crawling. Content already collected stays collected.
For bots that ignore it, use your CDN or server firewall. Some providers, including Cloudflare, offer one-click AI bot blocking.

Make AI search work for you

If you allow AI search bots, help them understand your site: clear headings, direct answers near the top of each section, and an llms.txt file that points to your most useful pages.

Questions

How do I block ChatGPT from crawling my site?

Add a User-agent: GPTBot group with Disallow: / to robots.txt to block training. Block OAI-SearchBot too if you also want to stay out of ChatGPT search.

Does blocking Google-Extended remove me from Google?

No. Google-Extended only controls use in Gemini models. Your pages stay in Google Search, including AI Overviews.

Should I block AI crawlers?

It depends on your goals. Blocking training bots protects content from model training; blocking AI search bots can cost you citations and referral traffic.

Do all AI crawlers respect robots.txt?

Major companies say their crawlers do. Some smaller or unknown bots don't, and need a firewall rule instead.

How to block AI crawlers with robots.txt