Locating Duplicate Pages

[Travis-W]

Be sure to try different combinations of 'site:www.domain.com' and 'site:domain.com'.  They will all yield different results.

Sounds to me like you probably have an internal search engine that is generating search results pages based off the search term, and each different results page is a piece of duplicate content.

[ahead4]

Does your website force 'www.'?

Both yourdomain.com and www.yourdomain.com are separate sites and can have different pages spidered.

[Asher]

it could be that your tags and categories are considered individual pages and therefore creating their own permalink: ex: http:www.example.com/keyword, and http://www.example.com/tag/keyword and http://www.example.com/category/keyword.  Another way would be to check the sitemaps you have in webmaster tools and compare those to each other.  Just a suggestion.

[MikeRoberts]

Have you checked for instances where a page parameter is being seen as another version of the same page? One of the sites I work for had an issue a few months back where every instance of a product page was being flagged as duplicate content because of an oversight. We had one of our coders write a clause into the page where every time a page loaded with a parameter such as ?color=72 it would canonicalize it to the page minus the parameter. This decreased our duplicate content warnings quickly and effectively.

[TommyTan]

Hi Chris,

Google Webmaster has a tool that helps identify duplicate HTMLs and maybe you can use that to see if the 11,000 pages are duplicate.  IF they are, I am assuming they should have the duplicate Title Tag and etc. which the tool may discover.

[ChrisHolgate]

Thanks for your reply.  Indeed our website does force www. if someone were to attempt to navigate to us without prefixing www.

[ChrisHolgate]

I'm not certain if this is the case as our search engine requires physical input in order to yield a result.  I don't know if it helps but the URL is http://bit.ly/4Cogchww if you fancy taking a look  

[ChrisHolgate]

We did have this at the beginning of the year when we used a ?dispmode=grid and ?dispmode=list to change the way our results were displayed.  This has been rectified however by us completely removing the option and any instances of dispmode present in the URL force a 301 to the correct master page.  There are still a few hundred instances of this dispmode being present in the Google index but 99% of them have fallen out now.

I have checked and double checked and we don't seem to have any issues like this at present.

[ChrisHolgate]

Many thanks for your response.  Our site is an eCommerce site that doesn't employ tags as such and our categories are all accounted for in the 15,000 page figure.

[ChrisHolgate]

Many thanks for your input on this.  I have actually looked at this through the HTML improvements section of GWMT however I am showing only a few dozen duplicated titles / descriptions and this is simply due to the product categories being almost identical (for example HP Deskjet 500 and HP Deskjet 500+)

[Travis-W]

I can't get that link to work.

What I said before still applies with physical input (this is what I assumed when I said it).

For example, user inputs the words "snakes and dogs" and clicks search.  The new URL is "www.yoursite.com/search?q=snakes and dogs"  All these weird URL pages need noindex meta tags or Google will flag them as duplicate content because, for example, this page and the result for "dogs and snakes" generate almost the same page.

Does that make sense?
It is in Google's Webmaster Guidelines that you should noindex these pages.

[ChrisHolgate]

Sorry, I'm not sure what happened to that bit.ly address - The actual address of the website is www.refreshcartridges.co.uk.

Ah, I see what you mean about the search results now however this hopefully shouldn't be an issue as for security (our web guy said something about injections) the URL that is returned irrespective of what is searched for is http://www.refreshcartridges.co.uk/advanced_search_result.php

Thanks again!

[Travis-W]

Hmm, I'm not too knowledgeable about php pages.  Sorry!

[ChrisHolgate]

It's cool.  Sorry, the point I was making is that irrespective of what you search for the page that is returned is http://www.refreshcartridges.co.uk/advanced_search_result.php (with nothing after the .php) and as such the search results page couldn't spurn multiple pages which could be indexed by Google.