Wild fluctuation in number of pages crawled

[MarkWill]

I am seeing huge fluctuations in the number of pages discovered the crawl each week. Some weeks the crawl discovers > 10,000 pages and other weeks I am seeing 4-500.

So, this week for example I was hoping to see some changes reflected for warnings from last weeks report (which discovered > 10,000 pages). However, the entire crawl this week was 448 pages.

The number of pages discovered each week seems to go back and forth between these two extremes. The more accurate count would be nearer the 10,000 mark than the 400 range.

Thanks.

Mark

[NewGlobalVentures.comSEOTexas]

Hi there Mark!

We used to have the same issue using Joomla here. It turns out that Google will reduce their crawling if your site has too many errors, broken links, and so on.

We used GWT to look into the 404's then redirected the broken links. Afterwards, we resubmitted the site to be reindexed. A few weeks later -VOILA- all is back to normal and our page freshness stays where it should.

I'd recommend looking at your GWT first, and fixing broken links followed by resubmission to SE's...

Good Luck!

[MarkWill]

Thank you for the response. I should have been more clear. It is the weekly SEOMoz crawl that is showing such inconsistent behavior, not Google. Sorry I wasn't more clear.

We have very few (if any) broken links, errors, etc.

Thanks.

Mark

[Cyrus-Shepard]

Hi Mark,

I used to troubleshoot these types of problems (mysteries when I worked on the SEOmoz help team.

The best thing to do would be contact the Help Team (help@seomoz.org) and include information both your account, url and campaign. They can take this information and see if there is anything odd about your website, or if there is a bug in the crawling software, or finally if there is some strange quirk of incompatibility causing this behavior.

If you would rather, you can PM me with the info and I can try to troubleshoot it myself, but the Help Team has a few more tools and access to engineers, so they might be the better choice. Either way, let us know if you have any trouble.

[NewGlobalVentures.comSEOTexas]

No problem!

Glad to see Cyrus' response!