Access denied in google webmaster tools
-
Hi I have just checked on my google webmaster tools and it is showing i 11 urls that are coming back as access denied. Now the urls are working, and they have been redirected using 301 redirect, so i have done everything right but for some reason google is not able to crawl them.
Does anyone know what i have done wrong for it to come back as access denied and how i can solve this problem. the site is www.in2town.co.uk many thanks
| | | |
| | 2 | Gardening/Gardening-Advice-What-is-Hydroponic-Gardening/menu-id-4991 | 403 | 4/11/13 |
| | 3 | Top-Showbiz-News/Super-Injunctions-Are-Right-Says-Hugh-Grant | 403 | 4/29/13 |
| | 4 | Entertainment-Tonight/Cheryl-Cole-wants-to-spice-up-The-X-Factor | 403 | 4/11/13 |
| | 5 | Tiger-Woods-paid-10000-a-time-for-sex | 403 | 4/20/13 |
| | 6 | Thousands-of-children-hurt-trying-to-stop-arguments-between-adults/Thousands-of-children-hurt-trying-to-stop-arguments-between-adults/menu-id-4448 | 403 | 4/24/13 |
| | 7 | News-Showbiz/Doctor-Who-changed-my-life-says-Matt-Smith | 403 | 4/29/13 |
| | 8 | The-Latest-Health-News/Hypnosis-Hypnotherapy-for-Relationships/menu-id-4744 | 403 | 4/11/13 |
| | 9 | Soap-Gossip-Latest-News/Emmerdale-Marks-bit-on-the-side-comes-to-home-farm/menu-id-4615 | 403 | 4/11/13 |
| | 10 | news/eastenders/ | 403 | 4/11/13 |
| | 11 | entertainment-news/Prince-William-Stag-Do-To-Be-Held-in-Cape-Town | 403 | 3/24/13 || | |
-
Hi Tim,
It looks like you have some sort of server setup that is trying to block dos attacks or similar. If you put your site into screaming frog after the first few pages it starts returning 403 errors (access denied). If you then look at a page in a browser your see the message: Forbidden Access (flooding).
Is it likely that this is happening when google is trying to spider the site also?
-
thank you for this, what would be the best way to solve this, as this must be affecting my rankings
-
contacted my hosting company and they said the following robots file could be blocking google and causing the 403 errors, i thought it looked standard to me, can anyone please have a look and let me know if my robots file is causing the problem
many thanks
If the Joomla site is installed within a folder such as at
e.g. www.example.com/joomla/ the robots.txt file MUST be
moved to the site root at e.g. www.example.com/robots.txt
AND the joomla folder name MUST be prefixed to the disallowed
path, e.g. the Disallow rule for the /administrator/ folder
MUST be changed to read Disallow: /joomla/administrator/
For more information about the robots.txt standard, see:
http://www.robotstxt.org/orig.html
For syntax checking, see:
http://www.sxw.org.uk/computing/robots/check.html
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/ -
they have also said my 301 redirects are causing the problems but i thought i had done this correctly
here are some of my redirects
The lines that are causing the issue are:
Redirect 301 /Jennifer-Aniston-upset-over-Brad-Pitt-Marriage /news/have-your-say/jennifer-aniston-upset-over-brad-pitt-marriage
Redirect 301 /In2town-Gossip/Liz-Hurley-Wants-Her-Husband-Back /news/have-your-say/liz-hurley-wants-her-husband-back
Redirect 301 /News-Celebrity/Take-That-and-Robbie-Williams-do-it-again /news/have-your-say/take-that-and-robbie-williams-do-it-again
Redirect 301 /Kevin-McCloud-does-not-like-the-word-Poverty /news/have-your-say/kevin-mccloud-does-not-like-the-word-poverty
Redirect 301 /Latest-Travel-News/Singapore-Tourist-Information-Singapore-a-must-for-Holidays/menu-id-4592 /news/holidays/singapore-tourist-information-singapore-a-must-for-holidays
Redirect 301 /Travel-Articles/Holiday-makers-are-rushing-to-buy-cheap-flights-to-Benidorm/menu-id-4998 /news/flight-news/holiday-makers-are-rushing-to-buy-cheap-flights-to-benidorm
Redirect 301 /The-Latest-Health-News/Stop-Biting-Your-Nails/menu-id-4744 /news/healthy-living/stop-biting-your-nails-with-hypnotherapy
Redirect 301 /Woman-celebrates-after-losing-weight-with-Weight-Loss-Hypnosis /news/gastric-band-hypnotherapy/woman-celebrates-after-losing-weight-with-gastric-band-hypnosis
Redirect 301 /Health-News-/-Stop-Smoking-Hypnosis-really-works-says-expert /news/health/stop-smoking-hypnosis-really-works-says-stop-smoking-expert
Redirect 301 /Animal-Health-News/Pet-Advice-Kennel-Cough-Advice-for-your-Pets/menu-id-4954 /news/dog-care/dog-kennel-cough
Redirect 301 /Travel-News/Flying-to-Australia-consumers-are-turning-their-back-on-Travel-Agents-over-cheap-flights/menu-id-4592 /news/flight-news/flying-to-australia-consumers-turn-their-backs-on-travel-agents-for-cheap-flights-to-australia
Redirect 301 /The-Latest-Health-News/Childbirth-Hypnotherapy/menu-id-4744 /news/health/childbirth-hypnotherapy
Redirect 301 /Travel-News/Brazil-Holidays-is-becoming-a-huge-hit-with-British-Tourist/menu-id-4592 /news/holidays/brazil-holidays-is-becoming-a-huge-hit-with-british-tourist
Redirect 301 /Celebrity-Gossip-Celebrity-News-and-latest-celebrity-gossip /Showbiz-Gossip
Redirect 301 /Latest-Travel-News/Travel-Magazine-reveals-secrets-to-finding-Cheap-Flights /news/flight-news/saving-money-on-flights
Redirect 301 /Lingerie-Brands-expert-says-Lingerie-improves-your-sex-life /lingerie-helps-improve-your-sex-life -
Hi Tim,
Your robots.txt looks ok from what I can tell. The 301s dont look odd (although what you have there is only one side of them right? I don't see the final page).
I think the clue is the message you get on the front end "Forbidden Access (flooding)". If you search for this phrase you start seeing references to the joomla SH404SEF module. See here for example: http://forum.joomla.org/viewtopic.php?p=1368937
I am not a joomla expert, but maybe it is a joomla issue instead of a server one, worth looking into.
-
hi both sides are there, for example from above,
Redirect 301 /Lingerie-Brands-expert-says-Lingerie-improves-your-sex-life /lingerie-helps-improve-your-sex-life
so i have the original page and then pointing to the destination page
i just do not understand after all the checks i have done why the error is happening
-
Hi Tim,
Well both those urls you give for the 301 are returning a 404, but I don't think they are the cause of your original problem which is the access denied issue. For that I am pretty sure you need to be looking at that joomla SH404SEF module.
-
going to go through these today as they may have changed since the update. so you feel the sh404sef could be causing the blocking problems, i will contact them.
-
just going into my webmaster tools and it says the following
| | Response Code | Detected |
| --- | --- | --- |<colgroup><col style="width: 45px;"><col style="width: 80px;"><col><col style="width: 120px;"><col style="width: 90px;"></colgroup>
| | 1 | Headlines-Celebrity-/Simon-Cowell-The-Wedding-Is-Back-On | 403 | 4/20/13 |
| | 2 | Gardening/Gardening-Advice-What-is-Hydroponic-Gardening/menu-id-4991 | 403 | 4/11/13 |
| | 3 | Top-Showbiz-News/Super-Injunctions-Are-Right-Says-Hugh-Grant | 403 | 4/29/13 |
| | 4 | Entertainment-Tonight/Cheryl-Cole-wants-to-spice-up-The-X-Factor | 403 | 4/11/13 |
| | 5 | Tiger-Woods-paid-10000-a-time-for-sex | 403 | 4/20/13 |
| | 6 | Thousands-of-children-hurt-trying-to-stop-arguments-between-adults/Thousands-of-children-hurt-trying-to-stop-arguments-between-adults/menu-id-4448 | 403 | 4/24/13 |
| | 7 | News-Showbiz/Doctor-Who-changed-my-life-says-Matt-Smith | 403 | 4/29/13 |
| | 8 | The-Latest-Health-News/Hypnosis-Hypnotherapy-for-Relationships/menu-id-4744 | 403 | 4/11/13 |
| | 9 | news/eastenders/ | 403 | 4/11/13 |
| | 10 | entertainment-news/Prince-William-Stag-Do-To-Be-Held-in-Cape-Town | 403 | 3/24/13 ||
when i have looked at more info on this it says the following
Access denied errors
In general, Google discovers content by following links from one page to another. To crawl a page, Googlebot must be able to access it. If you’re seeing unexpected Access Denied errors, it may be for the following reasons:
- Googlebot couldn’t access a URL on your site because your site requires users to log in to view all or some of your content. (Tip: You can get around this by removing this requirement for user-agent Googlebot.)
- Your robots.txt file is blocking Google from accessing your whole site or individual URLs or directories.
- Test that your robots.txt is working as expected. The Test robots.txt tool lets you see exactly how Googlebot will interpret the contents of your robots.txt file. The Google user-agent is Googlebot. (How to verify that a user-agent really is Googlebot.)
- The Fetch as Google tool helps you understand exactly how your site appears to Googlebot. This can be very useful when troubleshooting problems with your site's content or discoverability in search results.
- Your server requires users to authenticate using a proxy, or your hosting provider may be blocking Google from accessing your site.
i have asked my hosting company about this and they say everything is fine.
any help would be great to solve this
| |
-
As I said the clue is likely in the message you get from the front end "Forbidden Access (flooding)"
If you search for this, the results all seem to mention joomla and that module. If you look through those results there are some mentions of the security features of this SEF module and how to turn them on/off. It is impossible to say if this is 100% the cause of your issue, but if your hosting company say everything is fine, and the message shown is specific to this joomla module, then it is a likely candidate. All things being equal, try turning off this security feature and see if the access denied errors in GWT go away.
-
cheers for this, i have contacted the company to see what they say about this issue and hopefully it will be resolved.
-
02164
-
073.3k
-
05164
-
04614
-
05530
-
07941
-
071.2k
-
07995