Screaming frog Advice

[Andy-Halliday]

Hi

I am trying to crawl my site and it keeps crashing.

My sys admins keeps upgrading the virtual box it sits on and it now currently has 8GB of memory, but still crashes.

It gets to around 200k pages crawl and dies.

Any tips on how I can crawl my whole site, can u use screaming frog to crawl part of a site.

Thanks in advance for any tips.

Andy

[Angelos_Savvaidis]

Hey there mate,

Sorry to hear that you are having issues. You can actually ask Screaming Frog to use more RAM. If you haven't done that yet please give it a go.

You can find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/general/

If you want to crawl part of your site it can surely do that. You can exclude pages or whole sections.

Find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/

Hope this helps!

[RyanPurkey]

Hi Andy. There are quite a few settings you can adjust to make the server load less while the crawl is running. These can be found with descriptions here: http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/

For example, by not checking Images, CSS, SWF, and Javascript you'll be able to lessen load substantially, or if you'd like to crawl just a portion of the site you can set it to not check links outside of the start folder.

To have even more control over the crawl, you can use regular expressions to exclude certain pages, or sections that match a given pattern. The page above is fairly robust, so it should help you dial back the crawler to be friendlier to your server.  Cheers!

[Andy-Halliday]

Thanks, I tried all the tips on the screaming frog site, but I have just tried to 2 pages a second and lets hope that work.