Google haveing problems accessing part of my site

[ClaireH-184886]

hi my site is, www.in2town.co.uk and for a few weeks now google has had trouble accessing part of my site.

Today googlewebmaster tools tells me that google is having major problems it shows, 123 pages where access were denied.

i have spoken to my hosting company who could not find a problem, so not sure what to do now. can anyone please give me advice on what the problem may be.

any help would be great

[danatanseo]

Hi Claire,

I was able to view your robots.txt file but nothing else. I can't seem to get to the site. Is the URL correct?

Dana

[ClaireH-184886]

the site is www.in2town.co.uk     www.in2town.co.uk  can you try it with the link and without the link, not sure why you cannot get on the site but will be interested to find out.

this problem of google have access errors has been going on for some weeks now and i do not understand what is going on

[AlanBleiweiss]

One step I would suggest is that I would suggest using a tool like Screaming Frog. It can crawl your site and report on problems or broken internal links on the site.

I would also go into Google Analytics and go into the Content/Speed/Page Speed data. I would then expand the date range to cover from before they started listing problems until the day before you run the report (the current day can lag in updating in Google Analytics).

From there, look to see if there are any specific days where Google shows the page speed average (in the timeline chart) as being unusually high.  There may be days that are more problematic than others.

Finally, with that same date range in place, I would filter down the list of pages in the speed report down to show me one of the pages Google lists as being unreachable. Then do the same with some others.  See if Google shows any odd page speed times on those.

I would also then go to URIValet.com and run a check on some of those pages as well to see if URIValet shows the pages reachable and returning a proper "200" "okay" header message.

This is not a guarantee fix, however it can help bring answers.

[ClaireH-184886]

hi, thanks for this. i have used that tool and it has come up with as an example the following

http://www.in2town.co.uk/news/parenting/parents-need-to-act-against-child-obesity   403 forbidden

i am not sure why this has happened and this is the same result as i am being shown in google webmaster tools.

[AlanBleiweiss]

When I put that URL in, I get a "200" "OK" response.  If you get a "403 forbidden, you need to speak with the site systems (server) administrator to find out if they can pinpoint the problem.  Something is broken on the site at the server level to cause that.  Even if it's only intermittently broken.

[ClaireH-184886]

hi, when i put in http://www.in2town.co.uk  on the tool you told me about it brings up some pages that are fine and some that are 403, and the 14th one down is the url as above. that comes up with a 403 however, when i just put the url on its own, it comes up as a 200, which is strange.

Have you got any ideas on what i should say to the hosting company. i have spoken to them a number of times and all i get is, everything is fine but it is not

[AlanBleiweiss]

unfortunately I don't know exactly what else to suggest.  Who built the site?  Is it a custom site, or was it automatically built using a hosting company automated site builder?

If it's a custom site or you have a developer you've worked with to build it, speak with them. if it's a site automatically built using a hosting site building system, only they can help resolve it. If not, you'll need to get a different site built.  The problem you are describing should not be happening.

[ClaireH-184886]

the site has been built from a template but this problem has only started to happen in the past month and the site is a couple of years old

[AlanBleiweiss]

This confirms the need to get an expert involved - one experienced in these types of issues - a programmer or systems administrator type expert, not necessarily an SEO expert.

[ClaireH-184886]

i have asked about this problem on the google webmaster forum

http://productforums.google.com/forum/#!mydiscussions/webmasters/BogC5OZqdyM

and people are saying that it is a crawler issue with my hosting company,

they are suggesting a number of things that could be wrong which includes

Many hosting providers need to have mechanisms to throttle/limit either the number of requests or bandwidth to their customers sites.

However in this instance, they are not allowing for search engine crawlers which typically make numerous requests in quick succession. Significantly, their server's response is technically incorrect (RFC 2616), it should either not respond, or respond with a 503 (service unavailable). Google is aware of such incorrect 403s and will eventually try to crawl them again.To mitigate the issue somewhat, you can reduce your site's crawl rate in Google's Webmaster Tools

[AlanBleiweiss]

Fascinating how some problems can come up like this - it's not anything I'd ever seen - curious to see whether reducing Google's crawl rate will help.

[ClaireH-184886]

i have read it time and time again and do not understand how it can help. if you reduce the crawl rate then that means you reduce the number of pages and the number of times google visits your site or am i wrong

[AlanBleiweiss]

That's correct.  I don't know the specifics, however in this case, I would assume the reduced frequency/volume would mean less trouble dealing with the hosting provider's internal challenges.