Moz could not crawl my httpS website

[Hamedkhorasani1]

Hi,

we have a website with HTTPS, moz could not crawl it and we get "902 : Network errors prevented crawler from contacting server for page" while in logs we see moz robot access but fail after some seconds, what could be the problem, while moz can access site when it is without httpS

| 902 : Network errors prevented crawler from contacting server for page. |

[kevin.loesken]

Hello Hamed!

Here's a little more information from our Crawl Errors page:

_  902 Unable to contact server_

The crawler resolved an IP address from the host name but failed to connect at port 80 for that address. This error may occur when a site blocks Moz's IP address ranges. Please make sure you're not blocking AWS.

This question has come up a few times here in Q&A, so I'd recommend going to the following post for further help in getting this all diagnosed:
https://moz.com/community/q/odd-crawl-test-issues

I hope this helps. Thank you!

Kevin
Help Team

[Mobilio]

Did you block AWS IP range or Mozbot? Or did you applied some firewall policy?

PM me your URL so i can take a look on server and network level with different tools.

[Hamedkhorasani1]

Hi,

I have read those  and investigated, those are not reasons, because first server logs shows following:

<code>Line 31915: 2015-12-23 09:23:27 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /robots.txt - 80 - 54.167.7.207 HTTP/1.1 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 422 293 32

Line 31915: 2015-12-23 09:23:27 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /robots.txt - 80 - 54.167.7.207 HTTP/1.1 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 422 293 32

Line 31933: 2015-12-23 09:23:30 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET / - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 302 0 0 362 302 33

Line 31933: 2015-12-23 09:23:30 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET / - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 302 0 0 362 302 33

Line 31953: 2015-12-23 09:23:46 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /sitemap.xml - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 16407 313 74

Line 31953: 2015-12-23 09:23:46 W3SVC16 NEW-WEB-POUYASA 67.227.178.29 GET /sitemap.xml - 80 - 54.167.7.207 HTTP/1.0 rogerbot/1.1+([http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com](http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-starter-01@moz.com)) - - [www.pouyasazan.org](http://www.pouyasazan.org/) 200 0 0 16407 313 74

second crawl test shows following:
http://www.pouyasazan.org HTTP Status Code 302
httpS://www.pouyasazan.org HTTP Status Code 0

this shows that your crawler could connect

third, when i goto http://analytics.moz.com/search/on-page/4311859.1138194 i can optimiget results for keywork analys that shows your crawler could access my site</code> 

| http://www.pouyasazan.org |

[Hamedkhorasani1]

Hi,

website is httpS://www.pouyasazan.org

[GlennFerrell]

This post is deleted!

[Mobilio]

I check and seems that your site is OK from mine side - GCE, Amazon EC2 and in Eastern Europe. There are few issues - SSL3, weak DH and files as: https://www.pouyasazan.org/IT-news/25876/css/css/css/css/css/css/css/css/css/css/css/css/css/tooltipster.css (breaking relative and absolute path), also no 404 page because everything return "200 OK" (soft 404). But they isn't critical for your site working and it works. Server works and i just hammering it with many requests.

Please contact with Moz on mail here: help@moz.com or fill form there: https://moz.com/help/contact/pro to investigate your situation.