Google can't access/crawl my site!

[granitgash]

In mid-march website changed it's CMS but i don't think that could be the reason because until this week everything was working perfectly. I don't think it could have been compromised too. I'm still suspecting it could be the firewall blocking bots from crawling the site, but the server administrator couldn't find any evidence of this.

[Andy.Drinkwater]

It doesn't look like a firewall, as I can crawl it with Screaming Frog. However, the server logs will be able to answer that one for you.

Without looking in depth, I'm not seeing anything that stands out to me - do you think that there have been changes to the server that could cause issues? What firewall is the server running? Also, if there were errors in crawling the site, you would see a warning about this.

-Andy

[granitgash]

We are suspecting that CloudFlare might be causing these troubles. We are trying everything, in the meantime i'm looking here to see if anyone has any similar experience or an idea for solution.

As for warnings, the only warning we had was the one last week (8/23/14) saying that Google bot can't acces our site:

Over the last 24 hours, Googlebot encountered 316 errors while attempting to connect to your site. Your site's overall connection failure rate is 7.5%.

-Granit

[Andy.Drinkwater]

Ah OK - well keep us updated with what you find. Someone else will chip in with other info if they have some

-Andy

[Travis_Bailey]

A friend of mine just got back from Kosovo. It was the last stop on a tour of the Balkans. He had a pretty good time. Moving along...

I crawled about 12K URLs and hit almost 90 Internal Server Errors (500). It's probably not your core problem, but it's something to look at. Here are a few examples:

http://www.gazetaexpress.com/blihet/?search_category_id=1&searchFilter=1

http://www.gazetaexpress.com/shitet/?category_id=134&searchFilter=1

http://www.gazetaexpress.com/me-qera/?category_id=131&searchFilter=1

There was one actual page that threw a 500 at the time of crawl:

http://www.gazetaexpress.com/mistere/edhe-kesaj-i-thuhet-veze-22591/

The edhe kesaj page now resolves fine. (I'm not even going to pretend to understand or write Albanian.)

So there may be some issues with the server or hosting. If you haven't already, try this troubleshooter from Cloudflare.

[granitgash]

Hi Travis, thank you for your time.

Great for your friend, I also suggest to visit Kosovo someday, you will have great time here, for sure

Back to the issue:

Here is an interesting issue that is happening with the crawler.

Our own cms uses htaccess for rewrite purposes. I created 2 new files that are independent from CMS and tried to fetch them with WMT, and it worked like a charm.

These 2 independent files are:

www.gazetaexpress.com/test_manaferra.php

www.gazetaexpress.com/xhezidja.php

Then, I created an ajax page with our CMS, which contains only plain text, tried to fetch it by WMT and strangely enough it didn't work. To make sure that the .htaccess file is not affecting this behavior, I deleted the htaccess and tried to fetch it, but it didn't worked.

The ajax page is: www.gazetaexpress.com/page/xhezidja/?pageSEO=false

The site works perfectly for humans which access it via the browser.

I'm more than confused now!

ac857dfbf02a316d92d378bc48f9c395.png

[granitgash]

Hi all

Just wanted to let you know that we fixed the problem. We disabled CloudFlare which we found out was blocking Google bots. More about this issue can be found at: https://support.cloudflare.com/hc/en-us/articles/200169806-I-m-getting-Google-Crawler-Errors-What-should-I-do-

[KeriMorgret]

Great, thanks for letting us know what happened with this!

[Travis_Bailey]

This applies to the guy from Albania.

Oh, this IS the guy from Albania. Never mind.

[Travis_Bailey]

What did you do specifically to mitigate the problem? You can PM me, if you would like.