Sitemaps and Indexed Pages

[Slumberjac]

Hi guys,

I created an XML sitemap and submitted it for my client last month.

Now the developer of the site has also been messing around with a few things.

I've noticed on my Moz site crawl that indexed pages have dropped significantly.

Before I put my foot in it, I need to figure out if submitting the sitemap has caused this.. can a sitemap reduce the pages indexed?

Thanks

David.

TInSM

[eli.myers]

Hey David!

Thanks for reaching out to us!

Unfortunately I am not an SEO consultant / Web Developer so I cannot offer specific advice, but I'm sure there are loads of members here who would love to help and have a lot more knowledge than I do! A few things I have picked up which may help are the following:

Try to determine when the drop started, did it drop when you submitted the XML sitemap or when the developer changed certain things? This could help point to the reason for the drop in indexing. There are a variety of reasons as to why Google may not choose to index pages, however some of the common ones are:

• Check your robots.txt to ensure those pages are still crawlable

• Check for duplicate content / was there any canonical changes?

• One of the tools you could use to help keep track of ranking fluctuations is mozcast (http://mozcast.com/). Was there turbulence in the Google algorithm when the indexed pages dropped significantly?

If you want us to have a look at your specific campaign to investigate further could you please pop an email over to help@moz.com.

Thanks!

Eli

[Martijn_Scheijbeler]

Hi David,

Messing up, Changing or Updating, Deleting a Sitemap is not necessarily something that will decrease the number of ranked or crawled pages. It usually is used a signal to find new pages and figure out if old ones are deleted. But the chances that your sitemap have had a significant impact in what kind of pages went down is something I would find unlikely. It could happen though that you'd see the opposite, an increase in pages indexed/submitted/crawled after you submit a sitemap.

Martijn.

[Slumberjac]

Thanks Martijn!

I guess I was wondering if the MOZ Bot only followed pages that were in the sitemap. It was generated by Screaming Frog I have trusted it to include all relevant pages!

To elaborate.

There were about 80,000 pages and I used canonical, no index, and redirects to clean up a rather large mess of filter URL's and dup content.

That dropped the pages to about 14k. Then I submitted the sitemap last month and now the crawl only found 4k pages.

Further investigation is needed on my behalf but I wanted to double check that this sudden drop was not because of a sitemap! Thanks for clarifying that!

[Slumberjac]

Thanks Eli!

I guess I was wondering if the MOZ Bot only followed pages that were in the sitemap. It was generated by Screaming Frog I have trusted it to include all relevant pages!

I have put in a more detailed description in the response below. Overall I need to investigate further but i'm satisfied that the sitemap has not caused the drop!