Google Giving 404 Errors to Working Pages

[paulcorrado3]

We have a problem that just came up out of the blue with Google, in particular Webmaster Tools & our Merchant Center.  We're getting a 404 errors for 1000s of pages in both of these applications, but the pages are working just fine.  Our shopping campaigns have been hit hard as many of our top products have been suspended from our merchant feed.  Webmaster tools is giving us 1000s of 404 errors to product & category pages alike. But every page returning a 404 error is working.  We've contacted Google and they mention that they are not getting a 404 error, but their crawler seems to be.  Has anyone ever experienced this?  What could be causing this?

Thanks for any input.

[thewebguy3]

On Aug 16th i got a similar error  mydomain.com/page/feed/  is all giving 404 errors I look at the xml and it shows an xml  with title tags  it  has the site name - page not found. but the feed pages is not index in google.

[jeff-rackaid.com]

This could just be a WMT error.  I've seen this from time to time. If it persists for more than 24 hours, then you may want to try digging into it further.

Have you tried fetching as Googlebot

In doing server security work, I have seen some hijacking attempts that will send 301 responses to Googlebot.  In some cases, these scripts are improperly deployed resulting in 404 errors or 500 errors.  So I would first fetch the site as Googlebot.

Then examine your server's access logs for hits from Googlebot.  Confirm that it is indeed Googlebot.  You can verify Googlebot IP ranges with reverse DNS:

https://support.google.com/webmasters/answer/80553?hl=en

If there's no foul play, then check for any browser-based detections routines.  With the increasing use of mobile, I have seen some code fail to handle bots correctly.  The behavior is usually an error not a 404. However, many CMS systemincorrectly send 404 errors for a variety of errors and not just true not found status codes.

[paulcorrado3]

Thanks jeff-rackaid.com.

The idea of some sort of fowl play did cross our minds.  These errors seemed to appear out of the blue without any significant code changes on our side.

We have done a fetch as Googlebot on a number of pages.  We had found that the returns have been inconsistent; we would fetch a particular page and get a "Not found" error one minute and then a "complete" result the next minute.  I don't think this is a normal activity.

We've also been tracking the traffic of Googlebot to our site and since Thursday (8/14), the hits from GoogleBot have been going down while the hits from an "unknown bot" have been increasing.

[jeff-rackaid.com]

Yes, attackers use code injection techniques to hijack search results. They in fact do use some randomization techniques to make it harder to detect.  Conducting a security scan using a tool like Sucuri.net could be helpful but manually reviewing your .htaccess and code may be necessary.

[Christy-Correll]

Hi Paul, what is the status of this issue? Has it been resolved, or not? Would love an update! Thanks ~ Christy