Robots.txt blocking Moz

[ChristinaRadisic]

Moz are reporting the robots.txt file is blocking them from crawling one of our websites.

But as far as we can see this file is exactly the same as the robots.txt files on other websites that Moz is crawling without problems.

We have never come up against this before, even with this site.

Our stats show Rogerbot attempting to crawl our site, but it receives a 404 error.

Can anyone enlighten us to the problem please?

http://www.wychwoodflooring.com

-Christina

[ChristinaRadisic]

Our hosting provider has banned Rogerbot as they see it as problematic!!!!

They are a great hosting provider so this is going to be a difficult one.

[Nigel_Carr]

Hi Christina

I don't know how your site is set up but I can see that for some reason you are blocking access to the cgi-bin

If that directory contains files that execute php or other permissions then that may well be your problem. It's the only directory you are blocking and since I haven't seen other Robots.tx blocking it, then I would hazard a guess that this is the root of your problem.

Robots.txt

User-agent: *
Disallow: /cgi-bin/

Sitemap: http://www.wychwoodflooring.com/sitemap.xml

Regards

Nigel

[ChristinaRadisic]

Nigel,Thanks for the reply, the cgi-bin folder is never used by any of my sites but I put this in just as a matter of course, the folder would normally contain old cgi scripts so would not usually affect the crawling of a robot in any case.The reason for the problem turns out that our host had blocked rogerbot along with several other malicious bots, they have now lifted this block and the site is able to be crawled.- David

[Nigel_Carr]

Hi David

That's great news!

As a matter of interest, where did they block it? as it's not in the Robots.txt - was in in htaccess.txt?

Regards

Nigel

[ChristinaRadisic]

Hi Nigel

Neither, they use server side filtering.Regards- David