5XX (Server Error) on all urls

[GrumpyCarl]

Hi

I created a couple of new campaigns a few days back and waited for the initial crawl to be completed. I have just checked and both are reporting 5XX (Server Error) on all the pages it tried to look at (one site I have 110 of these and the other it only crawled the homepage). This is very odd, I have checked both sites on my local pc, alternative pc and via my windows vps browser which is located in the US (I am in UK) and it all works fine.

Any idea what could be the cause of this failure to crawl? I have pasted a few examples from the report

|

500 : TimeoutError

http://everythingforthegirl.co.uk/index.php/accessories.html	500	1	0

500 : Error

http://everythingforthegirl.co.uk/index.php/accessories/bags.html	500	1	0

500 : Error

http://everythingforthegirl.co.uk/index.php/accessories/gloves.html	500	1	0

500 : Error

http://everythingforthegirl.co.uk/index.php/accessories/purses.html	500	1	0

500 : TimeoutError

http://everythingforthegirl.co.uk/index.php/accessories/sunglasses.html | 500 | 1 | 0 |

Am extra puzzled why the messages say time out. The server dedicated is 8 core with 32 gb of ram, the pages ping for me in about 1.2 seconds. What is the rogerbot crawler timeout?

Many thanks

Carl

[Martijn_Scheijbeler]

Hi Carl,

You're a lucky man (sarcastic), your paging are loading just normally but are indeed giving the wrong status code: 500 for me. This is probably caused by some of the settings in Magento or you server as the normal status code for working pages should be a 200 OK.

That's probably also why Rogerbot didn't timeout on the pages but got a 500 while the pages were working. Good luck fixxing this!

Btw, I highly recommend using the Redirect Plugin for Chrome by Ayima.

[GrumpyCarl]

Thanks, will check out that plugin. So, in other words, the pages are loading fine for the user but sending out an error to the bots instead of the loaded ok message. That doesn't sound good!!

On the plus side, at least it has stopped Roger noticing some of the pages have up to 600 links on them because of all the retailer and manufacturer filtering options!!

Many thanks, Carl

[Martijn_Scheijbeler]

You're absolutely correct, hopefully this answered your question!