How can I prevent Google and other search engines to crawl my secured pages (https:)?

[esiow2013]

Let me know your thoughts guys. Thanks in advance!

[crackingmedia]

Hi esiow

You have a choice of placing a robots.txt file in the root folder of your website or if blocking individual pages you could use the meta robots tag. See these page for more information: http://moz.com/learn/seo/robotstxt and https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?csw=1

I hope that helps,

Peter

[Devanur-Rafi]

Hi there, blocking the HTTPS version of your pages from being crawled by the search engines is a bit tricky. You might need to come up with a separate robots.txt file to handle the HTTPS requests.

Here you go to know more about the process:

http://www.seoworkers.com/seo-articles-tutorials/robots-and-https.html

Hope it helps.

Best,

Devanur Rafi

[john4math]

Your best bet is to place a meta noindex tag on each secure page.  If it's only a few pages, you could just add it by hand.  If it's many, you should be able to access each pages protocol with whatever server-side language you're using, and dynamically add it on all secure pages.

If you use robots.txt to exclude the pages, Google can still show them in search results, with the description below them that reads, "A description for this result is not available because of this site's robots.txt – learn more."  Personally, I don't care for that.