Interest in optimise Google Crawl

[onibi29]

Hello,

I have an ecommerce site with all pages crawled and indexed by Google.

But I have some pages with multiple urls like : www.sitename.com/product-name.html and www.sitename.com/category/product-name.html

There is a canonical on all these pages linking to the simplest url (so Google index only one page). So the multiple pages are not indexed, but Google still comes crawling them.

My question is : Did I have any interest in avoiding Google to crawl these pages or not ?

My point is that Google crawl around 1500 pages a day on my site, but there are only 800 real pages and they are all indexed on Google. There is no particular issue, so is it interesting to make it change ?

Thanks

[PattyAMG]

Hi there!

From my experience, the best results I was ever able to achieve for a Client is when we consolidated all URLs to a single URL solution.  Canonicals are amazing, no doubt.  But I've experienced a canonical structure being ignored if there are instances where the canonical structure isn't 100% 'correct.'

If there is a way that you can have your website navigation & internal/XML sitemap reinforce your preferred URL, that would certainly reduce the number of URLs Google would crawl.  Then, if you permanently (301) redirect all the now non-navigable URLs to the single preferred URL, you should see a significant boost in traffic (from consolidating all of the authority into a single page, now reinforced throughout your entire website).

If that's not possible, and you have to have multiple URLs within your site for budget/platform constraints, then yes, let Google crawl them.  Otherwise the algo won't be able to see your canonical tag across them.

So in short: If you have a means to reduce the number of duplicates and redirect them - awesome. If you don't have a means to reduce duplicates, opening them up to Google is good, too.

For more information on making sure your canonical structure is set up properly, check out this Moz blog post: https://moz.com/blog/rel-confused-answers-to-your-rel-canonical-questions

[Kelly_Edwards]

Hi!

Have you no indexed the pages too? That may help to make sure that they aren't being crawled if that's concerning you. May at least give Google another signal not to crawl those pages.

Obviously it's not a catch all as there's only so much you can do to tell Google not to crawl a page. Sometimes if the alternative page is linked to internally (which it sounds like it is), then it will automatically crawl it even though you've said it has a canonical on it as you're showing that the page is important to your site.

May be worth testing a few pages to see if it has an impact.