How to find all indexed pages in Google?

[bjs2010]

Hi,

We have an ecommerce site with around 4000 real pages. But our index count is at 47,000 pages in Google Webmaster Tools.

How can I get a list of all pages indexed of our domain? trying to locate the duplicate content.

Doing a "site:www.mydomain.com" only returns up to 676 results...

Any ideas?

Thanks,

Ben

[ZephSnapp]

Use the tool Screaming Frog to see all your pages, that should help.  Also, the SEOmoz toolset has a function that will show you all duplicate content (if you are a pro subscriber).

[eyepaq]

Hi,

There are tools that you can use - though for close 50k pages is harder to crawl. Best bet is the Web master tools count - although is not 100% exact either.

The site:domain is a good indicator but it's generated "on the fly" but it will show you a better result if you go "deeper" and click on page 10-20 and so on.

However right now it looks like there is an issue with site:domain. for more info see: http://www.seroundtable.com/google-site-command-cluster-16829.html

Cheers.

[Francisco_Meza]

Hey Zeph! Hope your company is doing great.

@Ben, screaming frog is good for this. You will need to get the paid version of it. There is a video on the site http://www.screamingfrog.co.uk/seo-spider/. Use filters to get to your real URLs.

[bjs2010]

Thank you to everyone that contributed.

@Zeph and @Francisco - I do use Screaming Frog, but actually, correct me if I am wrong, but it does not show a list of pages indexed, but rather pages that exist in the site - not what Google has already indexed. Thanks anyway

What I wanted was a way of creating a list of all indexed pages in Google - not a count.

But thank you all the same!

[Francisco_Meza]

@BJS, I would export a file from GWT  and filter the results. If your URLs are in GWT, then most likely it's indexed in Google.

[ZephSnapp]

You are absolutely right.  But if you think that you have duplicate content issues, then Screaming Frog can help you tease that out.

That is also why I suggested the SEOmoz tool, since it is supposed to mimick a SE spider, it can give you a pretty good idea of any issues that you might have.

Using the advanced operator of site:domain makes sense, but if there are issues there like eyepaq said, it is going to be tough sledding.

My suggestion would be to download take a closer look at what GWT is telling you.  Are there duplicates there?  Is your CMS auto-generating URL's?  That is probably going to be your best bet IMO.

Best of luck!