Free SEO tool

Robots.txt tester

Enter a page address and pick a crawler. See if it's allowed, which line decides it, and test edits before you publish them.

How to use the tester

Enter the full address of the page you want to test and click Test URL. Clarnic loads that site's robots.txt, then shows whether the crawler you picked may visit the page and which line made the decision.

The robots.txt is loaded into the editor. Change a rule and the result updates as you type, for every crawler at once, so you can see the effect before you upload the new file. You can also paste a robots.txt that isn't live yet.

How crawlers pick a rule

A crawler first finds the most specific User-agent group that names it, and falls back to User-agent: * if none does. Inside that group, the rule with the longest matching path wins. If an Allow and a Disallow match with the same length, Allow wins. * matches any characters and $ marks the end of the address.

Google Search Console no longer has its own robots.txt tester, so this is a quick way to check a rule.

Questions

Why is a page allowed when I have Disallow: /folder?

Check the exact path and capital letters, and whether a longer Allow rule matches. Also check that the rule sits in a group that applies to the crawler.

Does a GPTBot group also apply to Googlebot?

No. Each crawler uses only the most specific group that names it. Rules under User-agent: GPTBot don't affect Googlebot.

Does it test query strings?

Yes. Include them in the address, like /search?q=shoes, to test rules that match parameters.

See what to fix on your website today.

Free plan, no credit card. Your first fixes in about a minute.