I’d like to set up a Moz campaign that crawls just the primary website, not subdomains
-
Is this something you could help with? (it either bombs or crawls everything, so I assume I’m missing something in the campaign settings, or it’s just not possible.
-
Hey, thanks for reaching out to us!
You can block our crawler from crawling specific subdomains of your site by excluding our user-agent from the subdomains robots.txt file. Our user agent is "rogerbot" and the directive would look something like this:
User-agent: rogerbot
Disallow: /You will want to make sure that this is associated with the robots.txt file specific to the subdomain you want to block.
You can read more about Rogerbot in our guide.
Hope that helps, let me know if you have further questions.