Google Index Status Falling Fast - What should I be considering?

[LauraSultan]

The canonicals appear to be set up correctly, and I would not advise listing the product URLs as their canonicals in the category as suggested above. That will create duplicate URLs with the same content, which is exactly what canonical tags are designed to avoid.

[LauraSultan]

A reduction in the number of pages indexed does not necessarily mean something is wrong. In fact, it could mean that something is right, especially if your rankings are improving.

How are you determining that only 80% of pages are indexed? Can you provide a specific URL that is not being indexed?

If you made changes to your canonical tag, robots.txt , or meta robots tag, these could all cause a reduction in the number of pages being indexed.

[ThomasHarvey]

What I've suggested will be avoiding these duplicate urls? Here's some actual examples, going via a tier two category I get the following product url:

https://www.symectech.com/epos-systems/customer-displays/pole-mounting-kit-94591.html

With a canonical of:

https://www.symectech.com/pole-mounting-kit-94614.html

Yet when going from https://www.symectech.com/epos-systems/?limit=32&p=2 (a tier 1 category) I get the canonical url.

So if there are products listed in multiple tier two categories then that's multiple urls for the same product. With the suggestion I made, there would only be one variation of this product url (the canonical)

[LauraSultan]

If I understand you correctly, you are referring to the following two URLs:

https://www.symectech.com/epos-systems/customer-displays/pole-mounting-kit-94591.html

https://www.symectech.com/pole-mounting-kit-94614.html

Both of these have the same canonical referenced, which is https://www.symectech.com/pole-mounting-kit-94614.html.

It doesn't matter what actually shows in the address box. For the purposes of indexation, what matters is what is referenced in the canonical tag.

.

[ThomasHarvey]

However, the canonical is only an advisory tag. I've had few cases where people have relied on their canonical tag when their site has numerous product url types (as above with category in the url and just product url) which has many references to these different urls elsewhere (onsite and offsite) and they are now indexed as both versions, which is not always ideal. It also means that reporting tools such as Screaming Frog only show the true URLs on the site. It's also saving crawl budget as it doesn't have to crawl the category produced url and the canonical url.

Whilst it's not a major issue, it's something I would look at changing.

[LauraSultan]

I would be interested in seeing examples of where this has happened. Were the canonical tags added after the URLs were already indexed or were the canonicals in place when the site launched?

[davebuts]

Hi Tim,

I agree with Laura on the canonical tags. I've worked on several large Magento sites and I've never seen any issue with the way Magento handles it - by canonicalizing product URLs to the root directory.

In fact, I actually prefer this was over assigning a product to a 'primary' category and using that as the canonical.

As Laura said, a reduction in the total number of indexed pages might actually be a really big positive here! More pages indexed does not mean it's better. If they are low quality/duplicate pages that have been removed from index, that's a really good thing.

I did find some issues with your robots.txt file:

• Disallow: /media/ - should be removed because it's blocking images from being crawled (this is a default Magento thing and they should remove it!)
• Disallow: /? - this basically means that any URLs containing a ? will not be crawled and with the way pagination is setup on the site, this means that any pages after 1 are not being crawled.
  This could be impacting how many product pages you have indexed - which would definitely be a bad thing! You would obviously want your product pages to be crawled and indexed.

Solution: I would leave Disallow: /? in robots.txt because it stops a product filter URLs being crawled, but I would add the following line:

Allow: */?p=

This line will allow your paginated pages to be crawled, which will also allow products linked from those pages to be crawled.

Hope this helps!

Cheers,

David

[Toby-Symec]

Thanks dude, I will take a look at this. Really appreciate you taking time to respond.

[Toby-Symec]

Thank you for taking so much time to look at our site. I really appreciate it. I will dig in to the points to see what we can achieve. Thanks again, Tim

[Toby-Symec]

Hi dude, thank you so much for taking time to look at this site. It is really kind of you. I will be taking a look at all the points raised over the next week to see what we can achieve. Thanks, Tim