Howcome Google is indexing one day 2500 pages and the other day only 150 then 2000 again ect?

[Zanox]

This is about an big affiliate website of an customer of us, running with datafeeds...

Bad things about datafeeds:

• Duplicate Content (product descriptions)
• Verrryyyy Much (thin) product pages (sometimes better to noindex, i know, but this customer doesn't want to do that)

[NakulGoyal]

Domain Authority / Pagerank is what Google looks to see how deep and how frequently Google will crawl a particular website. They also typically look into how frequently the content is being updated.

Think about it from Google's perspective. Why should they index that website, 2500 pages every day. What's changing ? Does the site have enough domain authority to warrant that kind of indexing ?

In my opinion, this is not a concern. Just submit XML Sitemaps and see what percentage of your submitted pages are indexed.

[danatanseo]

I agree with everything Nakul has said. Just to piggyback on that with additional information, try to think about it this way. Remember when someone gave you $1.00 when you were little and said "Don't spend it all in one place?" Well, someone at Google must have grown up with the same grandparents I did.

Okay, now, the analogy-free explanation

Google has a "crawl budget" every day. Every day that budget is allocated to millions of different sites. Now, by "sites" I mean "pages." Some pages change really frequently (i.e. the Yahoo New homepage). Some pages change hardly ever (i.e. an archived blog post). Also, some pages have very high PR and others, not so much. Also, some pages load extremely fast (consuming less of Google's bandwidth when the page is crawled) which leaves more Google resources available to Google to crawl more pages. Google likes it, and so should we all because people with fast sites are making it possible for everyone to get crawled more often (in essence, making them very considerate, well-behaved members of the Internet community).

So, based on all these, Google is going to apportion a part of its crawl budget to your site on any given day. Some days, it may have more room in its budget for you than others. Part of this might be effected by how fast pages, on any given day, load from your site. A ton of parameters can come into play here, including whether or not the pages on that day are heavier, or whether or not your servers are performing really fast on one day versus another.

I'd say the two things to be really concerned with after considering all of these things are:

1. Is Google indexing all of the pages you want indexed?
2. Is Google's cache date of your important pages recent enough? (i.e. 3 weeks or less)

If the answer is "no" to either one of those, then it's time to do some investigation to find out if there are technical issues or penalties that have been put in place that are hurting Google's ability or desire (not the right word to use about a bot, but I'm using it anyway) to crawl your pages.

Does that help?

[Zanox]

Hi Dana,

Thanks for your detailed explanation. Appreciate it   Off course I understand that site speed is a factor for crawling (+ ranking) and that the Google bots only want to spend a certain period of time on a website. It's more like, when servers are performing almost equal every day so page loads are igual to, what could it be?

I agree with your two points of considering, but I'm the type of guy that always wants to know why something is happening

@Nakul: Thanks for your responds!
The pages that are in and out of the index are mostly product pages. So the thing about "frequently updates" can be something. The website is pretty young so authority is not yet build as it should be for a big site. This can also be a factor cause the more authority the more time Google will spend indexing a website right

Anyway, great thanks for both of your answers!

Gr. Wesley