Decrease in index URLs, increase in URLs blocked by robots and drop in traffic

[pugh]

This post is deleted!

[CleverPhD]

Correlation is not always causation. As I read your GWT graph you have over 20,000 page indexed originally dropping to 10,000 pages indexed and you are talking about 7 pages blocked in robots.txt during that time period.

Do you use any robot no follow meta tags?  Did you implement any canonicals?  Are the pages that you blocked really important hub pages?  I just dont see how having 7 pages put into robots.txt would impact the spidering of 10K other pages otherwise.

Do you have a bunch of thin content on the site?  It may be that Google is dropping a bunch of low quality content.  If you have not seen any ranking drop for competitive terms then you probably need to look at your landing page report in GA and see if you can figure out if you do have landing pages that are having decreased traffic over that time period.  You can then look at those pages and see if they are still in the index (or not) and try and put all of this together.

My guess is that you probably have an idea of the "marginal" pages on your site.  They are of a certain type or a certain time frame when "things were done differently".  Use your advanced segment tool in GA to separate those out and see if they are a culprit.  If they are, consider getting rid of them (as Google is doing it anyway).

Also, I see you have a sitemap.  With 20,000 URLs plus, you could have an issue there and not even miss it.  I had a relaunch once and my IT team left out whole sections of the site. It is easy to miss when you have to go through 20K URLs

I am just brainstorming here - hope it helps!

[pugh]

Thanks for your reply.

Yes it does seem like a small number or URLs to be blocked but the correlations have made me consider it, although like you say that does not necesariy mean causation. It's just the drop in traffic has conincided perfectly with the decrease in indexed pages.

Do not have any robots meta pages. Do use some canonicals but I don't think it is that as they are used sparingly.

Some landing pages do have drops in traffic overtime but that is partly to be expected due to the product life cycle.

Do you think the sitemap could be causing problems? It is auto generated by pro-sitemaps.com Could it be better to just remove it altogether?

Thanks for the suggestions!

[MickEdwards]

I agree with CleverPhD to segment out suspect pages and review in analytics to see if there any patterns. Re robots perhaps there are newly generated URLs within /my-url(/)?

I'm dealing with a growing number of sites that seem to be failing in organic traffic for two distinct reasons -

On-site SEO is not tight, traffic has been good but not as good as it could have been and is now being highlighted by indirect residue fallout from Panda and Penguin which has had a negative effect on the link profile - everything starts to tail away.

I wouldn't concentrate on just robots this stage, rather do a complete audit on-site and on the link profile with an open mind.  Screaming Frog and of course the tools on here are a good place to start analysing.  Your drop in indexed pages may mean you have a tighter site which is the direction you want to go in - currently working on a site that had 900+k pages indexed in WT at one point now down to 15k and organic traffic is up 50%

[CleverPhD]

Thanks Michael for the additional input, agree completely - thumbs up.   Pugh, I think you are just being a little myopic on the whole situation and just need to start digging as your best answer here will be "it depends" most likely.

Regarding the sitemap and if you should remove it.  Have you even looked at the sitemap?  Have you audited it.  I would not use a sitemap unless I reviewed it regardless of who generated it.  On any enterprise sized site, you have to have automation for tasks.  Automations are based around assumptions and either they can be totally or partially wrong.  You then add in automation and then things can quickly and exponentially become a problem.  I have worked with awesome developers who have added a "feature" to "help", but then did not realize the issues it would cause. I have made some  dumb suggestions in my work as well - It happens to us all.

Just sounds like some roll up the sleeves, download some spreadsheet, sort and filter time!

[pugh]

Thanks for your suggestions chaps, I will take onboard what you have said and complete an audit.