MSNbot Issues

[tpt.com]

We found msnbot is doing lots of request at same time to one URL, even considering we have caching,  it triggers many requests at same time so caching does not help at the moment:

For sure we can use mutex to make sure URL waits for cache to generate, but we are looking for solution for MSN boot.

123.253.27.53 [11/Dec/2012:14:15:10 -0600] "GET //Fun-Stuff HTTP/1.1" 200 0 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

1.253.27.53 [11/Dec/2012:14:15:10 -0600] "GET //Type-of-Resource/Fun-Stuff HTTP/1.1" 200 0 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

1.253.27.53 [11/Dec/2012:14:15:10 -0600] "GET /Browse//Fun-Stuff HTTP/1.1" 200 6708 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

We found the following solution: http://www.bing.com/community/site_blogs/b/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx

Bing offers webmasters the ability to slow down the crawl rate to accommodate web server load issues.

User-Agent: *

Crawl-Delay: 10

Need to know if it’s safe to apply that.  OR any other advices.

PS: MSNBot gets so bad at times that it could trigger a DOS attack – alone! (http://www.semwisdom.com/blog/msnbot-stupid-plain-evil#axzz2EqmJM3er).

[OlegKorneitchouk]

Yes, you can add that to your robots.txt file and it should slow down the crawl rate. I haven't tested it myself but have seen many instances of it. Let us know how it works out!