Google indexing despite robots.txt block

[zeepartner]

100 points for you Martijn, thanks! I'm pretty sure you've found the problem and I'll go about fixing it. Gotta get used to having https used more frequently now...

[Martijn_Scheijbeler]

You're welcome, it was mostly due to noticing that the first snippet, the homepage, had no snippet and the rest of the pages did have one. That led me to looking at their URL structure. Good luck fixing it!

[Kingof5]

A noindex tag specific to Googlebot would also be a good idea.

[Kingof5]

People who are disagreeing with this, explain your reasoning.

[Martijn_Scheijbeler]

Did you mean a noindex tags for robots or a specific one for googlebot? With the second one I probably get the downvotes.

[Kingof5]

This post is deleted!

[Kingof5]

Specifically for Googlebot. I'm pretty surprised people would disagree - Stephan Spencer recommended this in a personal conversation with me.

[Martijn_Scheijbeler]

I thought that value was a bit outdated, turns out to be still accepted. Although it probably only address this issue for him in Google and I assume it will still remain one in other search engines.

Besides that the problem offered a way better solution in allowing Google not on the HTTPS site.

[zeepartner]

Yes, I think the crucial point is that addressing googlebot wouldn't resolve the specific problem I have here.

I would have tried adressing googlebot otherwise. But to be honest, I wouldn't have expected a much different result than specifying all user agents. Googlebot should be part of that exclusion in any case.

[john4math]

It sounds like Martijn solved your problem, but I still wanted to add that robots.txt exclusions keep search bots from reading pages that are disallowed, but it does not stop those pages from being returned in search results.  When those pages do appear, a lot of times they'll have a page description along the lines of "A description of this page is not available due to this sites robots.txt".

If you want to ensure that pages are kept out of search engines results, you have to use the noindex meta tag on each page.