What tool do you use to check for URLs not indexed?

[Syed1]

I usually just use Xenu's link sleuth (if you thousands of pages) to list out all the URLs you have and I would then manually check them, but I haven't come across an automated tool yet. If anyone knows any, I'd love to know as well.

[Hakkasan]

Curious about this question also, it would be very useful to see a master list of all URLs on our site that are not indexed by Google so that we can take action to see what aspects of the page are lacking and what we need for it to get indexed.

[kdaly100]

Saw this and tried the following which isn't automated but is one way of doing it.

• First install SEO Quake plugin
• Go to Google
• Turn off Google Instant (http://www.google.com/preferences)
• Go to Advanced search set the number of results you want displayed (estimate the number of pages on your site)
• Then run your site:www.example.com search query
• Export this to CSV
• Import to Excel
• Once then do a Data to columns conversion using ; as a delimiter (this is the CSV delimiter)
• This gives you a formatted list.
• Then import your sitemap.xml into another TAB in Excel
• Run a vlookup between the URL tabs to flag which are on sitemap or vice versa.

Not exactly automated but does the job.

[NakulGoyal]

I agree, this is not automated but so far, from what we know, looks like a nice and clean option. Thanks.

[ShaMenz]

Nice solution Kieran!

I use the same method, to compare URL list from Screaming Frog output with URL Found column from my Keyword Ranking tool - of course it doesn't catch all pages that might be indexed.

The intention is not really to get a complete list, more to "draught" out pages that need work.

[ezclickmedia]

I concur, Xenu is an extremely valuable tool for me that I use daily. Also, once you get a list of all the URLs on your site, you can compare the two lists in excel (two lists being the Xenu page list for your site and the list of pages that have been indexed by Google).

[lonniea]

David Kauzlaric has in my opinion the best answer. If google hasn't indexed it and you've investigated your Google webmaster account, then there isn't anything better out there as far as I'm concerned. It's by far the simplest, quickest and easiest way to identify a serp result.

re: David Kauzlaric

We built an internal tool to do it for us, but basically you can do this manually.

Go to google, type in "site:YOURURLHERE" without the quotes. You can check a certain page, a site, a subdomain, etc... of course if you have thousands of URLs this method is not ideal, but it can be done.

Cheers!

[Chris.McGiff]

I do something similar but use Advanced Web Ranking, use site:www.domain.com as your phrase, run it to retrieve 1000 results and generate a Top Site Report in Excel to get the indexed list.

Also remember that you can do it on sub-directories (or partial URL paths) as a way to get more than 1000 pages from the site. In general I run it once with site:www.domain.com, then identify the most frequent sub-directories, and add those as additional phrases to the project and run a second time, i.e.: site:www.domain.com site:www.domain.com/dir1 site:www.domain.com/dir2 etc.

Still not definitive, but think it does give indication of where value is.

[DanishWadhwa]

I think you need not any tool for this, you can directly go to google.com and search: Site:www.YourWebsiteNem.com Site:www.YourWebsiteName.com/directory I think this will be the best option to check if your website is crwled by google or not.

[squareplug]

hmm...I thought there was a way to pull those SERPs urls into Google docs using a function of some sort?

[Philip_Petrescu]

I would suggest using the Website Auditor from Advanced Web Ranking. It can parse 10.000 pages and it will tell you a lot more info than just if it's indexed by Google or not.

[rickv101]

Annie Cushing from Seer Interactive made an awesome list of all the must have tools for SEO.

You can get it from her link which is http://bit.ly/tools-galore

In the list there is a tool called scrapebox which is great for this. In fact there are many uses for the software, it is also useful for sourcing potential link partners.

[CMC-SD]

Unless I'm missing something, there doesn't seem to be a way to get Google to show more than 100 results on a page. Our site has about 8,000 pages, and I don't relish the idea of manually exporting 80 SERPs.

[Guest]

This post is deleted!

[CarlLarson]

This post from Distilled mentions that SEO for Excel plugin has a "Indexation Checker":
https://www.distilled.net/blog/seo/awesome-examples-of-how-to-use-seotools-for-excel/

Alas, after downloading and installing, it appears this feature was removed...

[Adlanera]

I generally just use Xenu's hyperlink sleuth (if you thousands of pages) to listing out all the URLs you have got and I might then manually take a look at them, however, see the guitar in demand I have not come upon an automatic device yet. If all people are aware of any, I'd like to recognize as properly.