Robots

[Paul78]

I have just noticed this in my code

name="robots" content="noindex">

And have noticed some of my keywords have dropped, could this be the reason?

[BenFox]

Could be.

That's a directive that tells search engines no to include that page in their indexes.

[BenFox]

Actually on second thoughts - YES. Yes it probably is the reason your terms are dropping.

[Paul78]

Thanks for your quick reply Ben.

It does not seem to be all my pages that have fallen off, just some, the developer said that it only got added this morning by mistake.

I actually typed in the full URL into Google and it does not appear anymore, I was ranked no.2 for that particular keyword, receiving about 150 click per day, not happy!

[zigojacko]

If you are using a content management system, these additional meta tags can often be controlled within your administration panel.

If the meta tag is hard coded into your website header, this will appearing on every page of your website and will subsequently result in you not having any pages indexed in search engines.

As Ben points out, the noindex directive instructs search engine robots not to index that particular page. It would recommended to address this issue as quickly as possible, especially if you have a high traffic website that is getting crawled frequently.

[Paul78]

Hi Geoff,

The developer had said it got added this morning when we rolled out a discount feature on our website, I think it was the CMS adding it automatically, however now a lot of the keywords that were ranking top 3 are no longer indexed, is it just bad luck? will Google come back?

[zigojacko]

Hi Gary,

If the pages dropped from Google's index that quickly, then chances are, they will be back again almost as quick. If your website has an XML sitemap, you could try pinging this to the search engines to alert them to revisit your site as soon as possible again.

It's bad luck that the meta tag was inserted and this caused immediate negative effects, but it will be recoverable, and likely your pages should re-enter the index at the same positions as they were prior to today.

The key is to just bring Google's bot back to your website as soon as possible to recrawl, publishing a blog post could do this, creating a backlink from a high traffic site (a forum is a good example for this) are some methods of encouraging this.

Hope that helps.

[Paul78]

Thanks Geoff, will do what you recommended.

I noticed in Google webmaster tools this:

Blocked URLs - 193

Downloaded - 13 hours ago

Status - 200 (success)

[Paul78]

Oh! I also noticed that in Webmaster tools that the sitemap was processed today, does that mean Googlebot has visited the website today?

[zigojacko]

Yes because that is what caused them to take notice of the meta noindex and drop your pages from their search results.

Best of luck with it, feel free to send me a PM if your pages haven't reappeared in Google's search engine over the next few days.

[Paul78]

The only thing that does not make sense to me is if the sitemap was processes today, why is the homepage still indexed?

[zigojacko]

Was the homepage one of the pages that included the noindex meta tag?

Even if it was, pages will not all be crawled at the same time or in a particular order. The homepage may have already been crawled before the change was made on your site, your homepage may not have even be crawled at all today if it was visited yesterday for example.

Crawling results can vary hugely based on a number of factors.

[Paul78]

It was everypage on the site.

I also noticed the pages that are not indexed no longer, they have no PR, is that expected?