Questions
-
How do I disallow crawl on a directory when it's a prefix to my site's URL?
Hey there! Tawny from Moz's Help Team here. You'll want to add a robots.txt file for that subdomain, and then add a Disallow command to that robots.txt file. So, using your example, you'd want a file like mediabank.mywebsite.org/robots.txt that had a Disallow command for any robots you don't want crawling that subdomain. For all user-agents, that would look something like this: User-agent: * Disallow: / That would stop any user-agents from crawling any pages on that subdomain. I hope this helps! If you've still got questions, feel free to send us a note at help@moz.com and we'll do our best to sort things out for you.
Other Research Tools | | tawnycase0