Three kinds of AI crawlers
AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) read pages so AI assistants can cite them in answers, with a link back. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) open a page because someone asked the assistant to. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot) collect pages to train future models.
Each is controlled separately in robots.txt, so you can, for example, stay visible in AI answers while opting out of training.
What this tool checks
It reads your robots.txt and works out, with the same rules the crawlers use, whether each one may crawl your homepage (or the page you entered). It also checks for an llms.txt file and shows which line decides each result.
What robots.txt can't do
robots.txt is a public request, and the large AI companies say they follow it. Fetchers acting for a user may not. Firewalls and CDN settings, like Cloudflare's AI bot blocking, can also block crawlers even when robots.txt allows them, so check those settings too.