Welcome to the Q&A Forum

I found confirmation from another thread that the Moz crawler has issues with SNI digital certs, which we use on our site. Here is an example of someone else who had the same issue:

https://forums.aws.amazon.com/message.jspa?messageID=614259

Note their closing comments:

"I turned off SNI and everything appears to work, as long as these services use the http version of the site. At the end of the day, nobody wants to take the blame. MOZ claims that SNI is supported but does admit that there appears to be some problem."

We can not "turn off SNI" and allow http access to our site, we must enforce https across all requests. Since Moz is unable or unwilling to engage with us further on this issue I suggest we find a different service with more up to date crawler software.

David:

We have found no issues in our logs so far, and I have simulated HTTP requests using the rogerbot useragent string with no problems. If there are no further responses from you regarding the questions I brought up in this thread having to do with wildcard SSL certificates I'm afraid I am out of paths to explore for why your software is unable to crawl our homepage.

John King

I asked the question about wildcard certificates because the problem started when we migrated the site to a new cloud host and that's when we switched over to using SANs (Subject Alternative Names) SSL certificates for *megapath.com and *.speakeasy.net. Is it possible that Moz might have cached old certs or IP addresses, or perhaps have trouble with SANs certificates?

question: Does the rogerbot spider ever have problems with wildcard certs? Because both www.megapath.com and www.speakeasy.net use "wildcard" digital certificates for https

One additional note:

• the 904 error means that the request timed out without a response from the server
• For any request, no matter what the useragent is, if the web server failed to give a response to an http request, that failure would be logged in the Fortigate firewall log and it would trigger a notification. that is part of the standard Fortigate functionality involved with healthchecks. Nothing like that has been logged or triggered notifications for our production cloud.

Additional notes for my previous post: The Fortigate doesn't detect by useragent, since it inspects only at the packet level. The reports were setup to detect by IP address and we have logging setup for a number of IP addresses that are associated with rogerbot according to this page: https://udger.com/resources/ua-list/bot-detail?bot=rogerbot#id10604

This is John King, the developer Randy spoke of in the previous comment. I asked our network engineer to setup firewall reports to log requests from the rogerbot useragent. So far we are unable to identify rogerbot-specific requests. The primary firewall sitting in front of the cluster in the cloud is a Fortigate device, and I have confirmed that there are no explicit rules configured to deny access or responses to requests coming from rogerbot useragent, or from any other useragent for that matter. The only hard and fast rule is that only port 80 and port 443 are open, and any request to port 80 (i.e. "http") gets a 301 redirect to port 443(i.e. "https"). Other than that there is no filtering going on, and all requests get a response. Even if that response is an error code like "500" or the like. If you are hitting the site via the URL "http://www.megapath.com" the response from the server should be a 301 redirect to "https://www.megapath.com". Same goes for our other sites like https://www.speakeasy.net", which Moz also seems to have a problem crawling.