What is the best tool to crawl a site with millions of pages?

[iCrossing_UK]

I want to crawl a site that has so many pages that Xenu and Screaming Frog keep crashing at some point after 200,000 pages.

What tools will allow me to crawl a site with millions of pages without crashing?

[YannickVeys]

For what purpose do you want to crawl the site?

A web crawler isn't really hard to write. In 100 lines of code you can probably code one. The question is of course: what do you want out of the crawl?

[iCrossing_UK]

Only basic stuff: URL, Title, Description, and a few HTML elements.

I am aware that building a crawler would be fairly easy, but is there one out there that already does it without consuming too many resources?

[McCannSEO]

Don't forget to exclude pages that don't contain the information you are looking for - exclude query parameters which just result in duplicate content, system files, etc. That may help to bring the amount down.