404s effecting crawl rate?
-
We made a change to our site where we all of a sudden we are creating a large number of 404 pages. Is this effecting the crawl/indexing rate?
Currently we've submitted 3.4 million pages, have over 834K indexed but have over and 330K pages not found. Since the large increase in 404s we've noticed a decrease in pages crawled per day. I found this Q & A in Webmasters (http://googlewebmastercentral.blogspot.com/2011/05/do-404s-hurt-my-site.html) but it seems like the 404s should not have an effect. Is this article out of date?
What do you think fellow Moz-ers? Is this a problem?
-
Yes i've seen this numerous of times. Is it just 404's are are there also things like DNS playing along?
But if the amount of 404's jump up really high then for sure google turns down the speed. i guess this gives you some air to fix it in time. -
That article you mention is very up to date.
but if you got "hit" by Google bot several times a day for those pages that now you return a 404 response code you will see a decrease in pages crawled per day since once Google sees a 404 response code it will not visit / hit that page that often aftre that... -
It's not a problem, just fix those as soon as you can. And yes, it does affect crawl rate from what I've seen.