Absurdly High Crawl Stats

[Tyler-Brown]

Over the past month and a half, our crawl stats have been rising violently. A few weeks ago, our crawl stats rose, such that the pages crawled per day worked out to the entire site being crawled 6 times a day, with a corresponding rise in KB downloaded per day.

Last week, the crawl rate jumped again, such that the site is being crawled roughly 30x a day. I'm not seeing any chatter at there about an algorithm change, and I've checked and double-checked the site for signs of duplicate content, changes in our backlink profile, or anything else. We haven't seen appreciable changes in our search volume, either impressions or clicks.

Any ideas what could be going on?

[TimHolmes]

Have you got anyone performing any monitoring on your site?

[KevinBudzynski]

Has the content change? Did you check crawl frequency in webmaster tools? Many issues can get this number up.

[Tyler-Brown]

No major changes to the content structure. Crawl frequency has always been set to google-optimized, and all the numbers i'm looking at are via crawl stats in GWT.

In terms of other monitoring on the site, we've got several tools running in regards to uptime/downtime, Moz Analytics, and a few other services that might crawl. I'm going to run Splunk on our server web logs this weekend to see what pages Googlebot is hitting.

[eyepaq]

Hi,

Are you talking about the crawl rate from Web master tools ? Can you share a screenshot inlcuding all 3 areas in the crawl section ? (time, kb and pages)

Cheers.

[KristinaKledzik]

Any feedback from your web logs on what Google's crawling?

[Tyler-Brown]

FWIW, after this crawl, our organic traffic has been gangbusters since this heavy crawl period (which stopped around 9/12.) Whatever this heavy crawl was, it appears to have been a net positive for us!