Moz Crawl Test error

[zoe.wilson17]

Moz crawl test show blank report for my website test - guitarcontrol.com. Why??? Please suggest.

[donford]

Hi Zoe,

I just tried to view your Robots.txt file and was unable to access it. So I requested a crawl. The error I got from Moz is "612 : Page banned by error response for robots.txt."

I would suggest you look at the file permissions of your robots.txt file. It should be readable by the public; Apache permissions would be (644). This means it is not available to view or it could be that it is in the wrong directory or non-existent altogether. It should be in your root and accessible via a browser here: http://www.guitarcontrol.com/robots.txt

I would also note that some people feel that listing your sensitive directories in the robots.txt file is a security risk, and perhaps they are right. However, there are other ways to secure a directory and keep them out of the Indexes, and if the information being protected is that important, I would imagine that all security precautions would be taken.

The robots.txt file is important for crawlers so it should always be used and available to be read. If security is the problem here, I would point to some high profile sites that don't seem to have a problem with showing the public their robots.txt file.
https://www.google.com/robots.txt
https://www.facebook.com/robots.txt

I hope this helps,

Don

<colgroup><col width="907"></colgroup>
| 612 : Page banned by error response for robots.txt. |