How can I fix this home page crawl error ?

[jlane9]

My website shows this crawl error =>

612 : Home page banned by error response for robots.txt.

I also did not get any page data in my account for this website ...

I did get keyword rankings and traffic data, I am guessing from the analytics account.

url =    www.mississaugakids.com

Not sure really what to do with this !

Any help is greatly appreciated.

[WilliamKammer]

This was brought up a little while ago, hopefully Chiaryn's answer here can help: http://moz.com/community/q/without-robots-txt-no-crawling

[jlane9]

Hello William, Thnaks for the heads up on that thread. I did see it, however my robot.txt file should be correct, so the answer in that thread did not help in my case. Looking for other options that could be the problem here. Cheers !

[WilliamKammer]

Yeah, your robot.txt seems fine, but the answer sounded like the error code could be misleading, so maybe you're looking in the wrong area for the root of the problem due to that reason.  Wish I could be of more help.

[DonnaDuncan]

Hi.

https://publib.boulder.ibm.com/infocenter/discover/v8r4/index.jsp?topic=/com.ibm.discovery.es.ad.doc/monitoring/iiysawhttp.htm says it's an error that occurred when the crawler attempted to connect to your Web server. It says a slow site or network might be the cause of the problem.

Your robots.txt is set correctly and your site is accessible (I just tried), perhaps your best bet is to wait and see if the problem recurs.

I also ran a scan of your site using screaming frog and got close to 2,000 internal server errors (response code 500) primarily from pages in this directory - http://mississaugakids.com/mississauga-events-calendar/. The pages are loading very slowly. That might be contributing to your problem.

Maybe start there and then circle back to see if the 612 error is recurring? I'm not very technical, but perhaps they're somehow related?

[jlane9]

Thank you Donna,

That's pretty much what I had found, but did not think that would cause the home page error I am seeing. I know that events calendar is a problem for speed. Maybe I will move the calendar off the domain and re scan. If the crawl is fine maybe I will move the calendar to a separate domain permanently.

[DonnaDuncan]

Or just noindex it for now? Seems worth a try unless someone more technical has a better suggestion.

Are you seeing the error only when Moz crawls the site? Google Webmaster Tools isn't signaling any problems?

D

[JaneCopland]

Hi there,

Is this still happening, or does it seem to have been taken care of?

Cheers,

Jane