Tool to Generate All the URLs on a Domain

[timfrick]

Hi all,

I've been using xml-sitemaps.com for a while to generate a list of all the URLs that exist on a domain. However, this tool only works for websites with under 500 URLs on a domain. The paid tool doesn't offer what we are looking for either. I'm hoping someone can help with a recommendation.

We're looking for a tool that can:

• Crawl, and list, all the indexed URLs on a domain, including .pdf and .doc files (ideally in a .xls or .txt file)
• Crawl multiple domains with unlimited URLs (we have 5 websites with 500+ URLs on them)

Seems pretty simple, but we haven't been able to find something that isn't tailored toward management of a single domain or that can crawl a huge volume of content.

[PatrickDelehanty]

Hi Felicia

Try ScreamingFrog - they crawl the entire site (you can configure how you want it to crawl your site) and have ways of creating a XML Sitemap for you.

The tool goes above and beyond those two areas as well and can do so much. I suggest you check it out! Good luck!

[SEO5Team]

2nd Vote for Screaming Frog. Tried a lot of tools to pull info on all the URL's and this tool is by far the best one for the job.

[sterlingbright]

It seems to crawl all the wordpress folders and media files. 
Is there not a tool that will tell you just your live website URLs, I'm after creating a site map and a mass re-organising content exercise, so want a list in excel of URLs.

Any tips welcome

Thanks

Sarah