Sitemaps and Indexed Pages

[Slumberjac]

Hi guys,

I created an XML sitemap and submitted it for my client last month.

Now the developer of the site has also been messing around with a few things.

I've noticed on my Moz site crawl that indexed pages have dropped significantly.

Before I put my foot in it, I need to figure out if submitting the sitemap has caused this.. can a sitemap reduce the pages indexed?

Thanks

David.

[ThompsonPaul]

No - submitting a sitemap won't reduce the crawl of a site. The search engines will crawl the sitemap and add these pages to the index if they consider them worthy. But they'll still also crawl any other links/pages they can find in other ways and index those as well if they consider them worthy.

Note though - having the number of indexed pages drop is not necessarily a bad thing. If removing a large number of worthless/duplicate/canonicalised/no-indexed pages cleans up the site, that will also be reflected in fewer crawled pages - an indication that quality improvement work was effective.

That help?

Paul

[Slumberjac]

Thanks Paul,

Yes there has been a big clean up of pages. There were over 80,000 to begin with. I managed to get that down to about 14k but then last month MOZ bot only crawled about 4,000 pages.

I was just a bit worried that the sitemap generated by Screaming Frog was incorrect and therefore that was the reason for the drop.

I was referring mainly to the MOZ site crawl. I guess I was worried that the MOZ bot only followed the sitemap!

There were loads of filter URL's and all sorts going on so it's a bit of a spiders web!

[ThompsonPaul]

Sorry - I missed the part about you looking specifically at the Moz crawler. While useful, it's a stand-in for what will actually be used for rankings - namely the actual crawls by the search engine crawlers themselves. I'd be looking right to the source for that info if you're concerned there's an issue, rather than trusting just Mozbot. You can find the SE crawlers data in Google Search Console and Bing Webmaster Tools. Look for trends and patterns there, especially around the sitemap report.

The challenge to a Screaming Frog-rendered sitemap is that it can only find what's linked. If the site has orphaned pages or an ineffective internal linking scheme, a crawl could easily miss pages. It's certainly better than no sitemap, but a map generated by the site's technology itself (usually the database) is safer.

P.