Can URLs blocked with robots.txt hurt your site?

[nicole.healthline]

We have about 20 testing environments blocked by robots.txt, and these environments contain duplicates of our indexed content. These environments are all blocked by robots.txt, and appearing in google's index as blocked by robots.txt--can they still count against us or hurt us?

I know the best practice to permanently remove these would be to use the noindex tag, but I'm wondering if we leave them they way they are if they can still hurt us.

[MichaelYork]

I don't believe they are going to hurt you, it is more of a warning that if you are trying to have these indexed that at the moment they can't be accessed. When you don't want them to be indexed i.e. in this case, I don't believe you are suffering because of it.

[MattAntonino]

I've seen people say that in "rare" cases, links blocked by Robots.txt will be shown as search results but there's no way I can imagine that would happen if it's duplicates of your content.

Robots.txt lets a search engine know not to crawl a directory - but if another resource links to it, they may know it exists, just not the content of it.  They won't know if it's noindex or not because they don't crawl it - but if they know it exists, they could rarely return it.  Duplicate content would have a better result, therefore that better result will be returned, and your test sites should not be...

As far as hurting your site, no way. Unless a page WAS allowed, is duplicate, is now NOT allowed, and hasn't been recrawled.  In that case, I can't imagine it would hurt you that much either.  I wouldn't worry about it.

(Also, noindex doesn't matter on these pages. At least to Google.  Google will see the noindex first and will not crawl the page. Until they crawl the page it doesn't matter if it has one word or 300 directives, they'll never see it.  So noindex really wouldn't help unless a page had already slipped through.)

[workzentre]

90% not, first of all, check if google indexed them, if not, your robots.txt should do it, however I would reinforce that by making sure those URLs are our of your sitemap file and make sure your robots's disallows are set to ALL *, not just google for example.

Google's duplicity policies are tough, but they will always respect simple policies such as robots.txt.

I had a case in the past when a customer had a dedicated IP, and google somehow found it, so you could see both the domain's pages and IP's pages, both the same, we simply added a .htaccess rule to point the IP requests to the domain, and even when the situation was like that for long, it doesn't seem to have affected them. In theory google penalizes duplicity but not in this particular cases, it is a matter of behavior.

Regards!