Why is our noindex tag not working?

[eddys_kap]

Hi,

I have the following page where we've implemented a no index tag. But when we run this page in screaming frog or this tool here to verify the noidex is present and functioning, it shows that it's not.

But if you view the source of the page, the code is present in the head tag. And unfortunately we've seen instances where Google is indexing pages we've noindexed. Any thoughts on the example above or why this is happening in Google?

Eddy

[DonnaDuncan]

I ran that page thru screaming frog and it came back with a "blocked by robots" status.

The second tool you suggested is not finding the noindex tag and I don't have an explanation for that, nor am I familiar with the tool.

A site command does not return any results.

Are you sure you have a problem? Is there another example you can provide?

[ThompsonPaul]

The example page does have a noindex tag in place, but it's not formatted correctly, so it's being ignored. Very subtle issue, but your tag is using "smart quotes" around the elements instead of the plain quotation marks that are required for code. If you look very carefully at the page source code, you'll see that they are quotation marks like you'd see in a Word document; the ones at the beginning of robots and noindex curl a different way than the ones at the end.) This usually occurs when the content was written in a word processor instead of a plain-text editor.

Because the tag's not formatted correctly, it's ignored by both the crawling tools and the search engines.

In addition, the site also has all pages blocked from crawling by the sitewide robots.txt file. This and noindex are conflicting instructions to search engines.

If a page is blocked in robots.txt, then the search engine will not crawl the page and so is not able to discover the noindex tag, even if it were formatted correctly. Therefore if the search engine becomes aware of the page in any other way than straight crawling (and there are a number of ways this can happen), then the page will still get indexed.

If it's a dev site, the proper way to keep it from being indexed is to either noindex all pages, or to put the site behind a password so the search engines and public visitors can't access it. If using noindex, the site must not be blocked with a robots.txt directive.

Does that all make sense?

Paul

[ThompsonPaul]

That SF response is from the robots.txt block, not a noindex tag though. SF is also ignoring the incorrectly formatted tag (as it should).

Paul

[davebuts]

Hi Eddy,

Edit: this was already answered before I could post my reply. But I've left the example.

The issue with the meta robots tag is that you are using curly quotation marks around robots and noindex:
You have:
“robots**” content=“noindex”/>
Instead of:
name="robots" content="noindex"**/>

This will fix your issue.

Cheers,

David