How to block my web from SeoMOz Crawler
-
Hi,
I want to block mysite.com/forum from SeoMoz Crawler, how do i do that? i just want to block the forum and let the other part of my site still crawlable by seomoz. Thanks
Regards
-
Howdy
You'll need to block the SEOMoz crawler in your robots.txt file. Edit the file and insert this bit of code:
User-Agent: rogerbot Allow: /
Disallow: /forum/If that doesn't work, you might want to add a Wildcard too, making it:
Disallow: *****/forum/
If you're not too sure about editing the robots.txt file, then it might be an idea to check the SEOMoz guide: http://www.seomoz.org/learn-seo/robotstxt - if you're really unsure, it's best to get someone else to do it for you, as you don't want to block all of your site to all of the crawlers
Hope this helps