How can I get a listing of just the URLs that are indexed in Google

[EvergladesDirect]

I know I can use the site: query to see all the pages I have indexed in Google, but I need a listing of just the URLs.  We are doing a site re-platform and I want to make sure every URL in Google has a 301.  Is there an easy way to just see the URLs that Google has indexed for a domain?

[modparent]

You may want to do that from an xml sitemap. You can find sites out there that will build a sitemap for you for free and then just open it in excel and you should have all of your urls in a list. NOw that doesn't answer your question of just the urls in google, but you will get all of the ones in google and then some if you do it the way suggested. Better overkill than underkill. Hope that helps!

[sferrino]

As Google will not show you everything, even using the site command, I use Yahoo SiteExplorer:

http://siteexplorer.search.yahoo.com/search?p=seomoz.org&bwm=i&bwmo=d&bwmf=s

and wrote a PHP script to take the TSV it exports and create a line for each page. I could probably make that available for use one one of my sites.

[EvergladesDirect]

Thanks, that let me grab the first 1000.

[sferrino]

Your welcome. If that fully answered your question please mark it as answered.

[jennita]

Ooh that would be great to let others use, maybe even a YOUmoz post?

[jennita]

A bit of a teaser... our new Firefox toolbar that's coming out soon will have the ability in the SERP overlay to download the page of SERPs

[sferrino]

Well here it is for those paying attention to this thread:

http://www.stevenferrino.com/scripts/redirect-parser.php

Not sure if posting a link will work, they tend not to for me, you can always copy and paste.

I'm considering the YOUMoz addition and already sent you an email Jennifer

[EvergladesDirect]

It didnt fully answer the question cause I was only able to get the first 1000 URLS.  I need to get the entire list.

[EvergladesDirect]

If you import the TSV into Excel you will get a column of just the URLS

[sferrino]

Ok, you haven't stated how big the site is. As I already stated, Google will not show you everything it has in it's index, Yahoo will give 1000, SEOMoz might have additional, also check your Google Webmaster Tools (if you have that setup).

The second thing to keep in mind is incoming links from other places. It sounds like there was no housekeeping before the restructure, so I would keep an eye on the web server logs, analytics, etc. and add 301's for anything else that comes in that doesn't exist.

It's not just about Google, it's also about the user experience. Going to a non-existent page can give the impression that whatever they are looking for is no longer mentioned on your website, which potentially looses customers.

[EvergladesDirect]

This question still remains unanswered, why did it get marked answered?