Questions
-
Rogerbot Ignoring Robots.txt?
I have just encountered an interesting thing about Moz Link Search and its bot: if you do a search for Domains linking to Google.com , you find a list of about 900 000 domains, among which I was surprised to find webcache.googleusercontent.com See the proof below in attache screen shot. At the same time, the webcache.googleusercontent.com policy for robots is as shown in the second attachment. In my opinion, there is only one possible explanation: Moz Bot does ignore robots.txt files... e9f7db874c 87ce35be1c
Moz Tools | | Mihas070