Website Server Issue?

[PrimeMediaConsulting]

I'm getting error messages that a website cannot be crawled and it might be due to the following issues:

Couldn't access the webpage because the server either timed out or refused/closed the connection before our crawler could receive a response. How to fix: Please contact your web hosting technical support team and ask them to fix the issue

Could Possibly Be:
1. DDoS protection system.
OR
2. Overloaded or misconfigured server

They asked me to talk to my hosting company about this issue and he's at a loss (I don't think he knows everything he needs to know potentially). Have you seen these issues before? Where is the best spot to start troubleshooting this issue?

[Martijn_Scheijbeler]

Hi,

Try using any other crawler to see if they provide you with the same message (ScreamingFrog), in some cases it could mean that the site is overloaded and will start blocking the crawler. We're doing something similar for ours as we have a lot of scrapers but exclude the 'good' sites.

Martijn.