Welcome to the Q&A Forum

Moz can't crawl his root domain (i tested using page grader) or any other URLs on his site,

So while the special chars might not help, i dont think thats the issue here :). Atleast, not the only one.

https://moz.com/researchtools/on-page-grader?keyword=preguntas&url=https%3A%2F%2Fwww.respuestasparaelhombre.com

Its possible your SSL is the issue, do you know if its an SNI cert? Moz crawler doesnt yet support SNI and a lot of recently issued certs are SNI ones...

Also, if your using Cloudflare, those certs are SNI as well.

http://serverfault.com/questions/506177/how-can-i-detect-if-a-server-is-using-sni-for-https

All reference i can find to this error is firewall issues. You will probably need to check with your hosting providers that they arnt blocking moz.

a similar post made last year:

http://moz.com/community/q/moz-tools-are-returning-url-is-inaccessible

DA on a couple of my accounts sank by 2 ~ 3 in the last update, maybe a minor update to the metrics triggered something...

A few more links about this, that i was able to dig out:

http://live.theverge.com/sundar-pichai-google-mwc-2015/

and the full thing:

http://www.mobileworldlive.com/mwl-extra-sundar-pichai-svp-of-products-at-google

Hi James,

Thanks, I know it's not a 1 day fix, it's just somewhat frustrating to find that Moz doesn't yet support technology that is fast becoming standard, and thats been around for years. I look forwards to hearing when Moz does have an ETA for this! The Crawler is the main reason I created a Moz account, so i can't emphasize enough how much of an effect not having a long term solution at least announced soon is going to have on my decision to remain a Moz subscriber in the long term.

If it makes any difference, turns out Bing doesn't support it either, so something else to remember as more people start seeing crawler errors / lack of rankings that they may not understand!

http://webmasters.stackexchange.com/questions/77267/bing-and-lack-of-sni-support

And thank you for confirming the issue :).

Examining the server logs yourself probably wont help your understanding of the issue unless you know what your looking at specifically. On the Yahoo note, i have found Slurp to be really bad in the past, but no legitimate bot should be able to bring down a properly configured web server, especially an 'enterprise-level' one.

I would check your .htaccess and apache settings for bad redirects (or web.conf if on windows) before considering banning the bot. Other things to check would be website code or if a bot hits a massive and horribly optimised Database Query for example, that could bring the server down.

Ask IT exactly what the bot did that caused the server to go down, they should atleast be able to tell you that. If not then they need to run load tests against the website itself to try and reproduce the scenario and thus debug the issue, if indeed there is one.

Tl;dr :- Normally bad config or code / queries are to blame for this kind of thing. I'd review that before blocking a bot that crawls hundreds of thousands of other sites without issue.

The main problem here is that i can't disable SNI unless i pay for a business account ($200 USD per website, per month), except for the enterprise account everything else is Pro which is more than enough for my clients (no way i could justify $200 a month just so moz can crawl).

SNI is a technology thats been supported since IE7 and is now standard in all browsers, so still not supporting it now is just a little insane....

But as i said, this is going to be effecting everyone using SSL under cloudflare (which is probably everyone using cloudflare) so could you guys please make this a priority? its not just an edge case issue.

Hi All,

Recently i've noticed that Moz can no longer crawl any of my wbsites that are behind cloudflare. After speaking to CF support, we have come to the conclusion that a recent change by CF to force SNI for all SSL domains is the issue.

As this is likely to affect almost everyone using Cloudflare could anyone please confirm this?

In the mean time, one of the websites currently blocked is on CF enterprise, so we've requested SNI be turned off. Will test crawl again when thats done to confirm SNI is the issue.

Since im here im going to respond to this with a couple of things.

"While Cloudflare offers many optimization services and DDoS protection, many of them break Magento and Wordpress functionality."

I've not had any issue with this. Typically if something does break in WP or Magneto its pretty simple to fix. Personally, i dont run WP sites without cloudflare, its too much of a security issue.

" Implementing caching and Google PageSpeed with Nginx or Apache shows much better performance,"

You should do this anyway....

" Worse though, Cloudflare doesn't allow you to use your own SSL on any plan under $200/moth"

Sure it does, i run 40 + sites on standard Comodo SSL certs on Cloudflare pro without issue.

"If you're a Pagespeed/YSlow nut, they you'll really be upset to learn Cloudflare adds a security cookie to everything, so there is no way to have serve content on a cookie-free domain when using Cloudflare DNS"

Thats a fair point, though perhaps not such an issue in most situations.

"Customer service is also slow and mostly automated, so don't expect great service on unpaid plans."

Again i dont really find this. Most of my accounts are CF Pro but for those that arn't, i dont notice any difference in support level. I know Business and Enterprise plans get faster support but for $200 + a month, i'd expect that anyway.

You may find that you have to wrap the code that gets called when Ajax fires in something to catch the user agent. I.e. if your making an Ajax request to a php script in order to return data, you could wrap that php code in something like this (please excuse the Sudo code):

if(in_array($_SERVER['HTTP_USER_AGENT'], $knownagents){

//known webspider, or blocked agent, return nothing.

return "";

} else {

//not a known spider so continue.

}

?>

Thats very generalised but you get the idea. I put a short list together in JSON format a while back, you can find it here if its of any use: https://www.source-control.co.uk/knownspiders/spiders.php

PM me if you need any more specific help than that with development, hopefully someone else will have a slightly easier way of dealing with this though heh

Nothing much i can add to Martijin's response, you might want to run a scan with Screaming Frog or Xenu to triple check, but 301's should be fine.

Ok, so, my view on this.

In response to livecam's comment, __VIEWSTATE (the code he was refering too) is a base64 encoded form field used in ASP.net to hold data. Its probably not malicous in this instance. see this: http://stackoverflow.com/questions/1350216/what-does-the-viewstate-hold

For me, when i search "prom dress shop" in an incognito chrome window, i dont see either entry on the front page of google, though i expect this is because im searching from the UK.

Reviewing the pages specifically, i can make a couple of suggestions.

1. Check your web.conf file, your main domain may have been hacked and this adjusted to send only search engine to um.org (to hide the hack)
2. it may be that um.org has used Black Hat SEO technique's to massivly raise its profile, this will be short term as google will slap them with loads of penalties pretty quickly.
3. Check your web server specifically for viruses etc. Being an ASP.net site, you'll be hosted on a windows server, running IIS. It will be just as prone to viruses as your windows PC at home (without the proper protection).

If you would like a hand to check your site code specifically, drop me a PM and we can see what we can do. Otherwise, if you have in house developers, they should be able to take a look.

Hi Jarno,

A couple of things to look at, did your site structure change when you switched to responsive? How has the responsiveness been done? (Poor implementation might cause you problems if google can no longer follow links correctly).

Are you testing while logged into a google account? Or logged out in an incognito window in Chrome? (The later will give you a much cleaner result as google adjust search results per user based on viewing habits). Or do you see these changes in Moz specifically?

Something to remember, mobile responsiveness has absolutally no known effect on desktop searches, only those carried out on mobile devices and so won't be a ranking factor for a large number of your keyword searches.

I would look at what going responsive changed though. If you like, PM me a link to the website (or link it here) and I’ll take a look to see if i can spot anything with the implementation.

Hi, Sorry i missed your response,

I think in general terms rather then me listing everything here, you should follow Patrick's advice as a start and come back once your done with all that as you will have a much improved base for further adjustment.

SEO is not a quick process, it can take months to get a good foundation if not longer, so keep that in mind when your making changes.

To add some more to this thread, and from a developer and technical point of view..

As everyone has pointed out, the site is horribly slow. I've dug into that a bit, including with a gtmetrix report which you can find here (its not good). http://gtmetrix.com/reports/www.sportzing.com.au/8kNVOOv2

As for your hosting, you seem to be hosted with Prompt Web Hosting, while i have no direct experiance with them, i notice that thier own site is loading very very slowly. I would seriasly suggest looking at a new provider with better server performance or atleast upgrading your current hosting plan.

The most important thing to remember with hosting is that you get what you pay for. The cheaper the package, the worse it will be in terms of speed, support, reliability and general service.

I also notice that your using an out of date Wordpress install, please remember to keep Wordpress up to date as not doing so leaves you more open to hacks etc.

You might also want to look at adding a service like Cloudflare to your website, it will help to speed things up a bit (as much as possible outside of getting a better server atleast) and provide an extra layer of security. All wordpress sites that we host we make sure have cloudflare, simply because how much of a liability Wordpress is for security.

"We went from Drupal to WordPress"

That seems like a very backwards move to me, that should be the other way around (My personal opinion anyway)

On the 404s though, any URL changes will cause problems, you will need to add in redirects to deal with those.

On a drupal specific note, if your old site was using the 'URL Redirect' module, you may have had a number of redirects in place in the database, these wouldn't have been in your .htaccess file and will no longer work after moving to wordpress.

I dont have a huge amount to add to this, other than in my experiance a lot of agencies tend to be 2 or 3 years behind in general practises, especially the smaller ones.

"The times of looking to rank numero uno for one term are long gone." is completely correct though, so be very careful about messing up other key words etc.

Being UK based, i can't go this year Boo!

I hope everyone else has fun though!

A couple of improvements i can suggest straight off:

You need to redirect all non-www traffic to www. or vice versa. Currently so far as google can see, you have 2 different websites, one on http://bassilimousineservice.com/ and one on http://www.bassilimousineservice.com/

Your URL structure needs to be improved,for example: http://www.bassilimousineservice.com/index.php/services/rates.html

Should be something like: http://www.bassilimousineservice.com/services/rates