Questions
-
How to authenticate Moz crawler so that others don't use Rogerbot useragent to scrape data from our site?
Hi There, Thanks for writing us so there seems to be a few things going on here so if you need any additional clarification please let me know. So Moz will use a dynamic IP, so there is not just one IP we can provide for authentication. Unfortunately, your best course of action in this case would be to authorize Mozilla/5.0 (compatible; rogerBot/1.0) This would need to be conducted on the hosting level so you would need to work with your current hosting provider for a viable solutions. Also, you could set up your robots.txt file to disallow all robots except for Google and Rogerbot, unfortunately malicious robots will often ignore robots.txt files, so any long term solutions would need to go through your hosting provider. I am sorry that we could not provide more assistance on this matter and hopefully the attacks on your site do not last. Have a great day!
Getting Started | | Sean_Peerenboom0