Blog page won't get indexed

[Happy-SEO]

Hi Guys,

I'm currently asked to work on a website. I noticed that the blog posts won't get indexed in Google. www.domain.com/blog does get indexed but the blogposts itself won't. They have been online for over 2 months now.

I found this in the robots.txt file:

Allow: /
Disallow: /kitchenhandle/
Disallow: /blog/comments/
Disallow: /blog/author/
Disallow: /blog/homepage/feed/

I'm guessing that the last line causes this issue. Does anyone have an idea if this is the case and why they would include this in the robots.txt?

Cheers!

[_nitman]

Hi Happy SEO,

Could you please share the blog URL here? Sounds like an interesting issue and would love to give a try to help you with this

[Happy-SEO]

Hi Nitin,

The URL is www.keukensduitsland.nl (/blog). The link to the blog page is in the bottom left corner called "Keukennieuws".

[_nitman]

Hi Happy SEO,

Well, the robots.txt looks find here. Could you try to fetch any of the blog page/post as google in the search console and share the screenshot here?

Also, to cross check the robots.txt (which looks fine though), you have robots.txt tester in search console where you can put any blog page/post to check if bots can crawl it. Please share a screenshot of that as well.

On a separate note, the sitemap.xml link mentioned in the robots.txt (http://www.keukensduitsland.nl/sitemap.xml) is broken. Fix that as well.

[DirkC]

It's not related to the robots.txt - you can easily check that in Webmastertools (Crawl > Robots.txt tester)

First issue is the location of the link - if you put a small link to the blog hidden in the left corner at the bottom of the page Google is not going to attribute a lot of importance to this link.

Most important issue on your blog articles is the canonical - example:

http://www.keukensduitsland.nl/blog/woontrends-2016/ has as canonical url: http://www.keukensduitsland.nl/woontrends-2016/ - however this page will redirect you with javascript to the blog article.

Make the canonical self referencing and do a proper redirect on the other pages (301 rather than js redirect)

Dirk

[DirkC]

To update - even worse: on the blog itself you are linking to the canonical version - not to the /blog/ version. So it would be impossible for Google to index /blog/ type of content.

If you do woontrends 2016 site:www.keukensduitsland.nl you will notice that the canonical version is properly indexed (even with the strange js redirect.

Dirk

[dohertyjf]

This is definitely the issue. Fix that canonical and they'll be indexed.

[_nitman]

Hi Dirk,

Good observation, I missed the canonical part somehow. So, google is indexing the canonical URLs here which doesn't have /blog/ in it and that's the problem. Have a look at the indexed page for this particular instance here. Non /blog/ instance is indexed, which will take you to its /blog/ version with wrong canonical URL.

Solution: Either remove the canonical URLs on these pages to point them to the current page itself. And yeah! As rightly mentioned by Dirk, do a proper /blog/ page linking from the blog page and other pages from where you're linking these articles.

[Happy-SEO]

Thanks alot!