Why do I have so many extra indexed pages?

[Tylerj]

Stats-

Webmaster Tools Indexed Pages- 96,995

Site: Search- 97,800 Pages

Sitemap Submitted- 18,832

Sitemap Indexed- 9,746

I went through the search results through page 28 and every item it showed was correct. How do I figure out where these extra 80,000 items are coming from? I tried crawling the site with screaming frog awhile back but it locked because of so many urls. The site is a Magento site so there are a million urls, but I checked and all of the canonicals are setup properly. Where should I start looking?

[Adriaan.Multiply]

As long as your organic traffic is doing fine I shouldn't be too concerned. That being said:

• Is your robots.txt or search console disallowing crawler access to parameters like '?count=' or '?color='?
• Is your robots.txt disallowing crawler access to urls that have a 'noindex' but were indexed before they got noindex?
• You can also take a couple of parameters from your site and test if any url's have been indexed, by using the 'inurl:parameter site:www.site.com' query.
• Are some of the canonicalized urls indexed anyway? This may indicate that page content is different enough for Google to index both versions.
• If there's a ton of articles that go in and out of stock and use dynamic ID's, Google may keep these in their index. Do out of stock articles return a 404 or are they kept alive?

[Tylerj]

I wouldn't say it is doing fine. Before I started they launched a new site and messed up the 301 redirects. Traffic hasn't recovered yet.

For Robots I am using the Inchoo robots.txt-http://inchoo.net/ecommerce/ultimate-magento-robots-txt-file-examples/ maybe it is a parameters issue, but I can't figure out how to see all my indexed pages.

I tried doing a search for both inurl:= site:www.site.com and inurl:? site:www.site.com and nothing showed up unless I am missing something.

I can't figure out how to check if some of the canonicalized urls are indexed. The pages are all identical though.

We have less then 100 out of stock items.

[Martijn_Scheijbeler]

• Have you checked out the parameters settings in Google Search Console to find out how many pages Google has found for your site with the same parameters? That might give some insights on that side.
• How many images do you have across the site? Do you have image sitemaps for these kind of pages.

What I would advise + what you've already been trying is to get a full crawl by either using ScreamingFrog or Deepcrawl. This will provide you with better insights into how many pages a search engine can really find.

[Tylerj]

1. Your first one is interesting. I actually haven't been in there before. There are 96 rows and everyone of them is set to let Googlebot Decide. Do you think I should change that up?

2. Not sure on how many images we have but it is a lot. Not we do not have an image sitemap.

I tried Screaming Frog and it couldn't handle it. After about 1.5 million urls it kept locking up. I just setup a free trial for Deep Crawl. It can only do 10,000 but I will see if it has anything worthwhile.

[ZestUK]

To ensure Screaming Frog can handle the crawl you could chunk up the site and crawl it in parts, e.g. by each subdirectory. This can be done within the 'configuration' menu under 'include'. There's loads of tutorials online.

You can also use exclude to ensure it doesn't crawl unnecessary pages, images or scripts for example on wordpress I often block wp-content

Definitely sounds like a problem with query parameters being indexed though and its often good to ensure these are addressed in the search console.

[Tylerj]

It ended up being my search results. I was able to use the site operator to break it down.