Can you stop/suspend a scheduled website crawl?

NC2

Hello,

If I do not want MOZ to crawl my site anymore in the future is there a way to turn this feature off?

Thank you

aarongray

Hey,

It's all done in Robots.txt.

This link is really useful when it comes to blocking someone from crawling a site (moz or anyone)

This one is also really helpful too

Hope this helps, also be careful when blocking a crawler if you make a mistake you can block everyone including google.

NC2

Thank you, I wasn't even thinking of this but that makes perfect sense. The 2nd link you gave me does not work though...can you verify it?

Do you specifically know what I could put in the file to block MOZ?

Thank you

aarongray

Yes the second link seems to be gone. Sorry about that

You need to add the following into robots txt. The crawler for Moz is called Rogerbot. The below will block the crawler.

User-agent: Rogerbot
Disallow: *

kevin.loesken

Hi Jeffery!

Check out this post to see how to go about it, as well as some other useful information about what to expect when blocking rogerbot:
https://moz.com/community/q/rogerbot-problems

I hope this helps

Thank you,

Kevin
Help Team

NC2

Thank you!

Welcome to the Q&A Forum