Site Spider/ Crawler/ Scraper Software

[AlexThomas]

Short of coding up your own web crawler - does anyone know/ have any experience with a good bit of software to run through all the pages on a single domain?

(And potentially on linked domains 1 hop away...)

This could be either server or desktop based.

Useful capabilities would include:

• Scraping (x-path parameters)

• of clicks from homepage (site architecture)

• http headers
• Multi threading
• Use of proxies
• Robots.txt compliance option
• csv output
• Anything else you can think of...

Perhaps an oppourtunity for an additional SEOmoz tool here since they do it already!

Cheers!

Note:
I've had a look at:

• Nutch
  http://nutch.apache.org/
• Heritrix
  https://webarchive.jira.com/wiki/display/Heritrix/Heritrix
• Scrapy
  http://doc.scrapy.org/en/latest/intro/overview.html
• Mozenda (does scraping but doesn't appear extensible..)

Any experience/ preferences with these or others?

[iPullRank]

Hey Alex,

Screaming Frog is hands down the best desktop crawling software and it has most of what you are looking for.

-Mike