Should we use Google's crawl delay setting?

[lzhao]

We’ve been noticing a huge uptick in Google’s spidering lately, and along with it a notable worsening of render times.

Yesterday, for example, Google spidered our site at a rate of 30:1 (google spider vs. organic traffic.)   So in other words, for every organic page request, Google hits the site 30 times.

Our render times have lengthened to an avg. of 2 seconds (and up to 2.5 seconds). Before this renewed interest Google has taken in us we were seeing closer to one second average render times, and often half of that.

A year ago, the ratio of Spider to Organic was between 6:1 and 10:1.

Is requesting a crawl-delay from Googlebot a viable option?

Our goal would be only to reduce Googlebot traffic, and hopefully improve render times and organic traffic.

Thanks,

Trisha

[adriandg]

Wow! those are really high render times.  Have you considered perhaps moving to another webserver? NginX is pretty damm fast, and could probably get those render times down.  Also, are you on a shared host? or is this a dedicated server?

What you're looking for is the robots.txt file though, and you want to add some lines like this:

User-agent: *
Disallow:
Crawl-Delay: 10

User-agent: ia_archiver
Disallow: /

User-agent: Ask Jeeves
Crawl-Delay: 120

User-agent: Teoma
Disallow: /html/
Crawl-Delay: 120

[adriandg]

more info on that here: http://www.robotstxt.org/

[lzhao]

We're a publisher, which means that as an industry our normal render times are always at the top of the chart.  Ads are notoriously slow to load, and that's how we earn our keep.  These results are bad, though, even for publishing.

We're serving millions of uniques a month, on a bank of dedicated servers hosted off site, load balanced, etc.

[RichardVaughan]

Unfortunately you can't change crawl settings for Google in a robots.txt file, they just ignore it. The best way to rate limit them is using custom Crawl settings in Google Webmaster Tools. (look under Site configuration > Settings)

You also might want to consider using your loadbalancer to direct Google (and other search engines) to a "condomised" group of servers (app, db, cache, search) thereby ensuring your users arent inadvertantly hit by perfomance issues caused by over zealous bot crawling.