Questions
-
Controlling crawl speed/delay through dynamic server-code and 503's
Hi INU, I always like avoid using things like 503s as a general rule. There is almost certainly a better way to do it. What about just using Google webmaster tools and Bing webmaster tools? Regarding HREFs it depends how much you rely on that tool. If you don't use it, then I'd more more likely to just block that bot in robots.txt and make sure Google and Bing are controlled using the appropriate tools in the respective webmaster tools. To answer your specific point about whether or not 503 can hurt rankings. In general no as long as they are only short-term. A 503 like 404s or any other response code is a natural part of the web, however, Google has said in the past that repetitive 503s can be treated as permanent rather than temporary and in some cases can result in the pages being removed from the index. I hope this helps, Craig
White Hat / Black Hat SEO | | CraigBradford1