How do I find which pages are being deindexed on a large site?

[DA2013]

Is there an easy way or any way to get a list of all deindexed pages?

Thanks for reading!

[matbennett]

If you have a full list of URLs you could check for cache date on each at Google.  Unless you were doing that manually it would be technically against google TOS, but so is SERP checking.  More to the point I don't think it would be foolproof as indexed pages will sometimes return no cache date.

It's a bit of a convoluted method, but I think that might be your only option.

[evolvingSEO]

Hi Daniel

Yep - as Mat says there's no official solution to this. Do you mean deindexed by Google (without you wanting them to be) or deindexed by you on purpose?

I suppose you could also;

• crawl your whole site
• depending how big the site is, do a site: search in Google.
• use the SERPs redux bookmarklet - get all indexed URLs in a column in a spreadsheet
• compare your crawl vs. the list indexed and whichever was not present in the SERPs could have been deindexed
• this method is faulty as it assumes all crawled URLs were indexed in the first place - but could get you part of the way there.

-Dan