In Index but not in Serps

[KMUK]

Hi,

I have a situation with a client site which is quite frustrating.

Basically, most "recent" (by that I mean for the last couple of months) blog posts are failing to reach the SERPS (actually, one has and a couple have from the early days but it's taken months for them to arrive).

Previously the blog posts were indexed very quickly - often instantly.

Now, I've checked WMT etc and I've submitted each post manually but still nothing.  The Sitemap is valid etc.

However, pages (not blog posts) seem to be getting into the serps very quickly.

Another complication is that if I search:

site:www.domainname.com and set the date filter to a month I can see some of the earlier blog posts in that result set.

However, if I scrape a bit of unique content from one of those posts and search - nothing in the SERPS.  And my Moz report tells me that the page is not to be found in the top 50 either (so I'm confident these pages are not in the SERPS).

Any ideas why this would happen to just blog posts?  Is it something to do with the parent blog landing perhaps being too strong in the rankings?

Any ideas appreciated.

Thanks.

[matbennett]

Really hard to say without knowing the site.  However my first place to look would be at technical issues. Are they crawl-able and indexable?

• Check robots text, metatags etc - are you allowing googlebot to crawl?
• Log in to webmaster tools and see if any URLs are being blocked/unreachable
• Point a crawler, such as Xenu or screaming frog at the home page. WIth all settings set to mimic Google, does it find the posts?

If it passes all of those it should be technically indexable.  If that doesn't turn up anything I would start looking at when Google is crawling the site (webmaster tools) and see if that is the problem.

[KMUK]

Thanks Matt - appreciate the reply.

Robots.txt are fine and I've check with other tools that the individual posts are not being blocked by robots.

WMT checks out fine too though the Crawl Rate is set to :  Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate.

Though the craw chart is indicating lots of crawling each day.

I do wonder about that crawl rate setting though.  The site is using CloudFlare for speed optimisation and I believe that is a common occurrence when using CloudFlare.

I'll check out Screaming Frog though.....good idea.

Thanks again.