Are there any negative side effects of having millions of URLs on your site?

[Deluxe]

After a site upgrade, we found that we have over 3.7 million URLs on our site. Many of these URLs are due to the facet options. Each facet combination yields a different URL. However, we need to do a deeper analysis into these URLs to see if this is the only reason why so many are returning.

Does anyone know if there are any negatives of having so many URLs crawled, other than the fact that Google only spends so much time crawling a site? Is the number of URLs something that should be concerning?

Any insight appreciated!

[Toddfoster]

There are several concerns to be addressed with this scenario:

1. Organization

This is going to be very difficult to keep track of. If you are well-organized or the pages will not need much adjusting, this is probably okay.

2. Duplicate Content

This is going to be a pain the behind. That being said, most site auditing tools will allow you to make adjustments as necessary.

3. Broken Links

With a site of this size, broken links and 404's are going to be inevitable. This could lead to some negative SEO impacts and will have to be kept on top of.

4. Hacking

This is a big reason why some sites have enormous numbers of URLs. This would likely be the biggest concern on my mind and worth looking in to. Going through that many pages will be impossible, so it might be worth taking a look at the link profile and determining where most of your links are coming from. If these are coming from spammy sites, you may have a problem there.

All this being said, the size of a website is normally not a cause for concern. Just make sure that your main pages (Home, Landing Pages) are properly handled and optimized and you shouldn't have too much trouble. I would add that unwieldy htaccess files (large ones) can result in slower loading times, which can impact your rankings with Google.

Let me know if there is anything specific concerning you and I will be happy to help. Congrats on the upgrade and hope it works out!

Rob

[MichaelC-15022]

I'll echo Robert's concern about duplicate content.  If those facet combinations are creating many pages with very similar content, that could be an issue for you.

If, let's say, there are 100 facet combinations that create essentially the same basic page content, then consider taking facet elements that do NOT substantially change the page content, and use rel=canonical to tell Google that those are all really the same page.  For instance, let's say one of the facets is packaging size, and product X comes in boxes of 1, 10, 100, or 500 units.  Let's say another facet is color, and it comes in blue, green, or red. Let's say the URLs for these look like this:

www.mysite.com/product.php?pid=12345&color=blue&pkgsize=1

www.mysite.com/product.php?pid=12345&color=green&pkgsize=10

www.mysite.com/product.php?pid=12345&color=red&pkgsize=100

You would want to set the rel=canonical on all of these to:

www.mysite.com/product.php?pid=12345

Be sure that your XML sitemap, your on-page meta robots, and your rel=canonicals are all in agreement.  In other words, if a page has meta robots "noindex,follow", it should NOT show up in your XML sitemap.  If the pages above have their rel=canonicals set as described, then your sitemap should contain www.mysite.com/product.php?pid=12345 and NONE of the three example URLs with the color and pkgsize parameters above.

[CleverPhD]

Agree with the points above with one exception.   Yes, you have to find a way to deal with duplicate and quality content at scale.  Yes, Robots.txt, nofollow links and index sitemaps are your friends.  I would not use rel=canonical unless I had to.  Better to get those extra pages de-indexed and then not let Google crawl the urls with the extra parameters to start with.  Why waste Google's time in crawling pages that are just resorted versions of another?   If you use the directives wisely you probably "only" have 200,000 pages worth crawling if you have that many sort parameters.

Good luck!