Blocked by robots.txt: What It Means and How to Fix It

In Google Search Console's Pages report, "Blocked by robots.txt" means Googlebot tried to crawl an address but a Disallow rule in your robots.txt told it not to. You'll also sometimes see a related warning: "Indexed, though blocked by robots.txt", which means Google indexed the page anyway, from links, without being able to read it.

Test which rule blocks a page

Enter the blocked address in the tester on this page and choose Googlebot. It loads your robots.txt, shows the exact rule that matches, and lets you edit the file and test again before you change anything on your server.

Is it a problem?

Check what the blocked pages are.

Blocked pages	Action
Admin, cart, checkout, account pages	Fine, leave them blocked
Internal search results, filter combinations	Usually fine, often intended
Product, service, blog or category pages	Problem: fix the rule
CSS, JavaScript or image files	Problem: Google needs them to render pages
The whole site	Urgent: likely a `Disallow: /` left from staging

How to fix an unwanted block

Find the rule. Use the tester to see which Disallow line matches the address.
Edit robots.txt. Remove the rule, narrow it, or add an Allow for the path you want crawled. Remember: the most specific (longest) matching rule wins.
Upload the new file to the root of your site. On WordPress, Shopify and other platforms, robots.txt may be generated by a setting or plugin rather than a file.
Ask Google to refresh it. In Search Console's robots.txt report (under Settings), you can request a recrawl of the file.
Validate the fix in the Pages report so Google rechecks the affected addresses.

Examples

A rule that blocks more than intended:

User-agent: *
Disallow: /p

This blocks /private/, but also /products/, /pricing and every other path starting with /p. Use the full folder with a trailing slash: Disallow: /private/.

Opening one path inside a blocked folder:

User-agent: *
Disallow: /resources/
Allow: /resources/guides/

"Indexed, though blocked by robots.txt"

This happens when you block a page Google already knows about from links. Google can't see the content, so it may show the URL with no description. If you want the page out of Google, unblock it in robots.txt and add a noindex tag. Google needs to crawl the page to see the noindex. If you want it in Google, just unblock it.

Prevent it from happening again

Check robots.txt after every site launch, migration or platform change.
Never copy a staging site's robots.txt to the live site.
Keep rules specific, with full folder paths.

For a full explanation of how rules work, see what is robots.txt.

Questions

What does blocked by robots.txt mean?

A Disallow rule in your robots.txt prevents Googlebot from crawling that address, so Google can't read the page's content.

Is blocked by robots.txt bad?

Only if the blocked pages are ones you want in search. Blocking admin, cart and account pages is normal.

How long does it take Google to see a robots.txt change?

Google usually rechecks robots.txt within a day. You can request a recrawl in Search Console's robots.txt report.

How do I remove a page from Google if it's blocked by robots.txt?

Unblock it, add a noindex tag, and let Google recrawl it. Google must be able to crawl a page to see the noindex.

"Blocked by robots.txt": what it means and how to fix it