Sitemap Indexed vs. Submitted

[moon-boots]

My sitemap has been submitted to Google for well over 6 months and is updated frequently, a total of 979 URLs have been submitted by only 145 indexed. What can I do to get Google to index them all?

[LoganRay]

Hi,

I poked around a bit on your sitemap and noticed a couple things:

1. You've got URLs on there that have canonicals to another page. For example:This page https://www.interstellarstore.com/meteorite-jewelry/meteorite-necklaces has a canonical tag that points here https://www.interstellarstore.com/meteorite-necklaces.
2. A bunch of the URLs in your sitemap redirect elsewhere or have no response - I got 13% through crawling your XML sitemap with Screaming Frog and there were zero 200 response code URLs, not good.

Both of these things combined are causing a discrepancy in the amount of submitted URLs vs. indexed URLs. If you use Screaming Frog to create your XML sitemap it's quite easy to have only clean URLs in there. You can easily remove all URLs that are not 200 status and by default Screaming Frog will exclude any URL that canonicalizes to another URL.

Also, as a side note, you should be updating your XML sitemap more frequently, a 6 month old sitemap for an ecommerce site is far too old with new products being added and products dropping off.

[moon-boots]

Hi Logan,

I tried using Screaming Frog but it kept finding useless links, so I wrote the sitemap myself and I update it manually, I updated it only this morning. What makes you think it is over 6 months since an update?

I was told on Moz in an earlier post that having all of the category links, not just the canonical ones, wasn't a problem, is this not the case?

Every link in the sitemap should work fine, I wrote it by copy and pasting the links directly from my site. I have no trust in Screaming Frog.

[LoganRay]

SF finding 'useless' links is actually part of its purpose, if you believe they're useless you should be asking why they're there. Your XML sitemap should have nothing but clean URLs; 200 response codes and not canonicalized to another URL. The problem isn't that you have category URLs, it's that those (like the one in my previous example) have a canonical tag that points elsewhere. Anytime this is the case, the URL is considered un-indexable. You can see the proof of this by doing a Google search for "https://www.interstellarstore.com/meteorite-jewelry/meteorite-necklaces", I just checked and this URL isn't in the index.

You mentioned the age in your original comment that your XML sitemap had been submitted for well over 6 months, that's where I got the age from, maybe I misunderstood?

You have no reason to not trust SF, it's one of the most valuable tools in an SEO's toolbox. I've used it for 5+ years to create hundreds of sitemaps and countless other SEO tasks with no problem in providing reliable, accurate data points.