Increase in pages crawled per day

[RyanPurkey]

There could be several factors... maybe your brand based search is prompting Google to capture more of your site. Maybe you got a link from a very high authority site that prompts higher crawl volumes. Queries that prompt freshness related to your site could also spur on Google. It is a lot of guesswork, but can be whittled down some by a close look at Analytics and perhaps tomorrows OSE update (Fresh Web Explorer might provide some clue's in the meantime.) At least you're moving in the right direction. Cheers!

[MickEdwards]

I would also check you have not got a spike of URL parameters becoming available.  I recently had a similar issue and although I had these set up in GWT the crawler was actively wasting its time on them.  Once I added to robots the crawl level went back to 'normal'.

[AFW1179]

When you say URL variables do you mean query string variables like ?key=value

That is really good advice. You can check in your GWT. If you let google crawl and it runs in to a loop it will not index that section of your site. It would be costly for them.

[MickEdwards]

yes, I updated to parameters just before you posted

[max.favilli]

I am not sure I understand what you mean, that website has a total of 35k pages submitted through sitemap to GWT, of which only 8k are indexed. The total number of pages indexed have always been slowly increasing through time, it moved from 6k to 8k in the last couple of months, slowly with no spikes.

That's not the total number of pages served by the site, since dynamics search results page amount to around 150k total pages, we do not submit all of them in the sitemap on purpose, and GWT shows 70k pages as the total number of indexed pages.

I analyzed Google crawler activity through server logs in the past, it does pick a set of (apparently) random pages every night and does crawl them. I actually never analyzed what percentage of those pages are in the sitemap or not.

Internal link structure was built on purpose to try to favor ranking of pages we considered more important.

The point is we didn't change anything in the website structure recently. User generated content have been lowering duplicate pages count, slowly, through time, without any recent spike. We have a PR campaign which is increasing backlinks with an average rate of around 3 links per week, and we didn't have any high DA backlinks appearing in the last few weeks.

So I am wondering what made google crawler start crawling much more pages per day.

[max.favilli]

Hi Ryan,

• GWT (Search Traffic->Search Queries) shows a drop of 6% in impressions for brand based searches (google trends shows a similar pattern).
• GWT is not showing any recent backlink with an abnormally high DA.
• we actually had a couple of unusually high traffic from Facebook thanks to a couple of particularly successful post, but we are talking about a couple of spikes of just 5k visits and they both started after the rise of pages crawled per day.

If you have any other idea it's more than welcome, I wish I could understand the source of that change to be able to replicate it on other websites.

[max.favilli]

Tw things I forgot to mention are:

1. something like 2 weeks ago we turned the website responsive, could it be google mobile crawler is increasing the number of crawled pages, I have to analyze the logs to see if the requests are coming from google mobile crawler
2. the total number of indexed pages didn't change, which make me wonder if a rise in the number of crawled pages per day is all that relevant

[RyanPurkey]

Ah, the responsive change could be a big part of it. You're probably getting crawls from the mobile crawler. GWT wouldn't be the best source for the recency on backlinks. I'd actually look for spikes via referrers in Analytics. GWT isn't always that responsive when reporting links.  Still, it looks like the responsive redesign is a likely candidate for this, especially with Google's looming April 21st deadline.

[max.favilli]

I usually analyze backlinks with both gwt and ahrefs, and ahrefs also doesn't show any abnormally high DA backlink either.

Agree the responsive change is the most probable candidate, I have a couple of other websites I want to turn responsive before April 21st, that's an opportunity to test and see if that is the reason.

[RyanPurkey]

Agreed. Especially since Google's own Gary Illyes respond to the following with:

How long is the delay between making it mobile friendly and it being reflected in the search results?

Illyes says “As soon as we discover it is mobile friendly, on a URL by URL basis, it will be updated.

Sounds like when you went responsive they double checked each URL to confirm.  From: http://www.thesempost.com/googles-gary-illyes-qa-upcoming-mobile-ranking-signal-change/. Cheers!

[AFW1179]

Nice find Ryan.