How to authenticate Moz crawler so that others don't use Rogerbot useragent to scrape data from our site?

[longclimber]

Is there any way to authenticate genuine Moz crawler. Because, our website keeps getting scrapping attacks and if there is no way to authenticate Moz crawler, then, any scraper can just set user agent as Rogerbot and scrape all our pages.

Is there a fixed IP that can be used or any other customization that will help us authenticate and allow only Moz crawler to crawl our site.

Looking forward to a solution to this problem. We haven't been able to use Moz crawler due to this issue.

[Sean_Peerenboom]

Hi There,

Thanks for writing us so there seems to be a few things going on here so if you need any additional clarification please let me know. So Moz will use a dynamic IP, so there is not just one IP we can provide for authentication.

Unfortunately, your best course of action in this case would be to authorize Mozilla/5.0 (compatible; rogerBot/1.0) This would need to be conducted on the hosting level so you would need to work with your current hosting provider for a viable solutions.

Also, you could set up your robots.txt file to disallow all robots except for Google and Rogerbot, unfortunately malicious robots will often ignore robots.txt files, so any long term solutions would need to go through your hosting provider.

I am sorry that we could not provide more assistance on this matter and hopefully the attacks on your site do not last.

Have a great day!