Sitemap generators for large sites?

[William.Lau]

What sitemap generators are you guys using right now? I'm in need of a generator that is not allowed to be installed onto the server. So perhaps an external crawl and generator.

The site has A LOT of pages in the millions.

[KevinBudzynski]

I use gsitecrawler for years with some large sites. However, not for millions...

[William.Lau]

I actually came across that everywhere else but wasn't sure. Site was a bit outdated and last time software was updated in 2007.

How long does it take to crawl pages for you?

I may have been exaggerating about millions but more like 2 million pages.

[KevinBudzynski]

Yes, it's pretty fast and multithreaded. It's old--but the job is pretty simple (multi threaded is the key, running many crawlers). At 2 million pages, probably will be incredible slow--but can do a site w/10,000+ pages pretty fast.

[William.Lau]

Thanks, I will try this out when I get to my Windows PC.

If anyone else has a good external crawler please do post.

[mark_baird]

xenu link sleuth is the on I use from time to time. It's a pretty handy tool.

[KevinBudzynski]

and blazin' fast!

[William.Lau]

Wish I could try it right now but won't be able to yet. Screaming Frog I believe does sitemaps and my site crashes it.

[simon_realbuzz]

I use Inspyder. We have a pretty big site containing lots of user generated content and it seems to manage the task pretty well. The only difficulty I have is understanding their rules of how to set which folders to ignore so I have to resort to manual removal of some URLs after the main crawl.

[PopTop]

I use the DYNO Mapper sitemap generator for large sites, but it is probably not best for millions of pages.

watch?v=kjFBnQCoxjI