Could large number of "not selected" pages cause a penalty?

[theLotter]

My site was penalized for specific pages in the UK On July 28 (corresponding with a Panda update).

I cleaned up my website and wrote to Google and they responded that "no manual spam actions had been taken".

The only other thing I can think of is that we suffered an automatic penalty.

I am having problems with my sitemap and it is indexing many error pages, empty pages, etc... According to our index status we have 2,679,794 not selected pages and 36,168 total indexed.

Could this have been what caused the error?

(If you have any articles to back up your answers that would be greatly appreciate)

Thanks!

[Marcus_Miller]

Hey Sarah

There are always a lot of moving parts when it comes to penalties but the very fact that you lost traffic on a known panda date really points towards this being a Panda style of penalty. Panda, is an algorithmic penalty so you will not receive any kind of notification in Webmaster Tools and likewise, a re-inclusion request will not help, you have to fix the problem to resolve the issues.

The not selected pages are likely a big part of your problem. Google classes not selected pages as follows:

"Not selected: Pages that are not indexed because they are substantially similar to other pages, or that have been redirected to another URL. More information."

If you have the best part of 3 million of these pages that are 'substantially similar' to other pages then there is every change that this is a very big part of your problem.

Obviously, there are a lot of moving parts to this. This sounds highly likely this is part of your problem and just think how this looks to Google. 2.6 million pages that are duplicated. It is a low quality signal, a possible attempt at manipulation or god knows what else but what we do know, is that is unlikely to be a strong result for any search users  so those pages have been dropped.

What to do?

Well, firstly, fix your site map and sort out these duplication problems. It's hard to give specifics without a link to the site in question but just sort this out. Apply the noindex tag dynamically if needs be, remove these duplicates from the sitemap, heck, remove the sitemap alltogether for a while if needs be till it is fixed. Just sort out these issues one way or another.

Happy to give more help here if I can but would need a link or some such to advise better.

Resources

You asked for some links but I am not completely sure what to provide here without a link but let me have a shot and provide some general points:

1. Good General Panda Overview from Dr. Pete

http://www.seomoz.org/blog/fat-pandas-and-thin-content

2. An overview of canonicalisation form Google

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=139066

3. A way to diagnose and hopefully recover from Panda from John Doherty at distilled.

http://www.distilled.net/blog/seo/beating-the-panda-diagnosing-and-rescuing-a-clients-traffic/

4. Index Status Overview from Google

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=2642366

Summary

You have a serious problem here but hopefully one that can be resolved. Panda is a primarily focused at on page issues and this is an absolute doozy of an on page issue so sort it out and you should see a recovery. Keep in mind you have 75 times more problem pages than actual content pages at the moment in your site map so this may be the biggest case I have ever seen so I would be very keen to see how you get on and what happens when you resolve these issues as I am sure would the wider SEOMoz community.

Hope this helps & please fire over any questions.

Marcus

[matbennett]

Yes, this can definitely cause problems.  In fact this is a common footprint in sites hit by the panda updates.

It sound like you have some sort of canonical issue on the site: Multiple copies of each page are being crawled. Google is finding lots of copies of the same thing, crawling them but deciding that they are not sufficiently unique/useful to keep in the index.  I've been working on a number of sites hit with the same issue and clean up can be a real pain.

The best starting point for reading is probably this article here on SEOmoz : http://www.seomoz.org/learn-seo/duplicate-content  . That article includes some useful links on how to diagnose and solve the issues as well, so be sure to check out all the linked resources.

[theLotter]

Hi,

Thanks for the detailed answer.

We have many duplicate pages, but they all have canonical tags on them... shouldn't that be solving the problem.  Would pages with the canonical tag be showing up here?

[Marcus_Miller]

Canonical tag to what? Themselves? Or the page they should be? Are these pages unique by some URL variables only? If so, you can instruct Google to ignore specific get variables to resolve this issue but you would also want to fix your sitemap woes: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=1235687

This is where it gets sticky, these pages are certainly not helping and not being indexed, Google Webmaster tools shows us that, but if you have this problem, how many other technical problems could the site have?

We can be almost certain you have some kind of panda filter but to diagnose it further we would need a link and access to analytics to determine what has gone wrong and provide more detailed guidance to resolve the issues.

This could be a red herring and your problem could be elsewhere but with no examples we can only give very general responses. If this was my site I would certainly look to identify the most likely issues and work through this in a pragmatic way to eliminate possible issues and look at other potentials.

My advice would be to have the site analysed by someone with distinct experience with Panda penalties who can give you specific feedback on the problems and provide guidance to resolve them.

If the URL is sensitive and can't be shared here, I can offer this service and am in the UK. I am sure can several other users at SEOMoz can also help. I know Marie Haynes offers this service as I am sure Ryan Kent could help also.

Shout if you have any questions or can provide more details (or a url).