Can i exclude paths from my crawl?
-
My site is setup like this mysite.com with mysite.com/blog
The blog portion is overwhelming my SEO efforts - I'd like to exclude that from seomoz reporting? is there anyway to do that in the campaign manager? or will I have to modify my robots.txt to inform roger not to crawl there?
-
Here's SEOMoz's Article on the subject, authored by Rogerbot himself:
http://www.seomoz.org/dp/rogerbot
You just need to add this to your robots.txt:
<code>User-agent: rogerbot Disallow: /blog</code>Should do the trick.