The noindex tag: what it does and when to use it

A noindex directive tells search engines: you can visit this page, but don't include it in your results. It's one of the most useful controls you have, and one of the most dangerous, because a single line left on by mistake can remove a page (or a whole site) from Google.

Two ways to set noindex

The robots meta tag

Placed in the page's head:

<meta name="robots" content="noindex">

You can target one crawler by name, for example <meta name="googlebot" content="noindex">.

The X-Robots-Tag header

Sent by the server as an HTTP header. This works for files that have no HTML head, such as PDFs and images:

X-Robots-Tag: noindex

Both are treated the same way. The header is easy to miss because it doesn't show in the page source, which is why checking with a tool matters.

When to use noindex

• Thank-you and order confirmation pages.
• Internal search results pages.
• Login, account and cart pages.
• Thin tag or archive pages that add nothing to search.
• Staging and test sites (better still, protect them with a password).
• Duplicate printer-friendly or campaign landing pages.

Noindex and robots.txt don't mix

This is the most common mistake. If robots.txt blocks a page, Google can't crawl it, so it never sees the noindex tag. The page can stay in the index, sometimes shown with no description. To remove a page from results, allow crawling and use noindex. Once it has dropped out, you can leave it that way.

Google also stopped supporting noindex written inside robots.txt in 2019. It must be in the page or the header. See what is robots.txt for what robots.txt can and can't do.

Noindex, nofollow and other values

index, follow is the default, so you never need to write it.

How accidental noindex happens

• A "discourage search engines" setting left on after a site launch (WordPress has one).
• A staging site's settings copied to the live site.
• An SEO plugin rule applied to a whole post type.
• A CDN or server rule adding X-Robots-Tag: noindex to every response.
• A page builder template with noindex switched on.

If traffic to a page drops suddenly, check for noindex first.

How to check if a page is noindexed

Enter the address in the checker on this page. It reads the robots meta tags, the X-Robots-Tag header and robots.txt, and tells you in one line whether Google can index the page and why. In Google Search Console, the Pages report lists every page "Excluded by 'noindex' tag", which is a good place to find pages you didn't mean to exclude.

How long until a noindexed page disappears?

Google has to recrawl the page to see the tag. Important pages can drop out within days; rarely crawled pages can take weeks. To speed it up, use the URL Inspection tool in Search Console and request indexing, or use the Removals tool for a temporary hide.