Mac-Friendly, DOM-Rendering Spidering Tool for Multiple Users

[mirabile]

Hello!

I am looking for a spidering tool that:

• Is Mac-friendly
• Can render the DOM and find JS links
• Can spider password-protected sites (prompts for password and then continues spider, etc.)
• Has competitive pricing for 8+ users.

Screaming Frog is amazing - and maybe we're just going to have to bite the bullet there.  But if anyone has any other ideas, I've love to hear them.  Thanks!

[Martijn_Scheijbeler]

Going to follow this, as I've been looking for something too. But we went the cloud service, as there is nothing that I acme across that can otherwise fulfill all these needs.

[mirabile]

Oh, interesting - can you help me understand about more about the cloud solution are you using...? Thanks!

[Martijn_Scheijbeler]

Oh actually something I just realized is that potentially ScreamingFrog can do what you want and it will provide you with access to 8 users, but the setup is complicated. You would need to run it in a big virtual machine on AWS or Google Cloud Platform. That way you can scale the machine so it won't time out and everybody will still have access to it.

Back to your question: I've worked with Deepcrawl, a bit with Ryte and more with Botify. They're all great tools that are able to crawl your site. But you probably already looked into some of them.

[Mobilio]

Right now we're updating SEOSpyder ( http://www.mobiliodevelopment.com/seospyder/ ) for rendering pages but i can't give you timeframe when will be done.

So far memory requirements isn't too high and was crawl 250k site with 8G ram machine.

[mirabile]

So - after digging around a lot and reading and re-reading every article that popped up for "screaming frog alternative", I've come to the conclusion that for the price, there really is nothing better than Screaming Frog right now.

I was impressed, however, with the incredibly helpful team from Deep Crawl.  This enterprise tool is designed for larger websites - whereas Screaming Frog can crap out of your local machine runs out of memory.  Because it's a more powerful tool, it's more expensive than Screaming Frog - but if you need an enterprise solution, it's definitely worth looking into.  Another big differentiator is that Deep Crawl has no limit to the number of users, which is our primary pain point with Screaming Frog.