Duplicate content and http and https

[mediabase]

Hello Hawkvt1, Fisrt of all I want to tell you that  the protocols (http/https) are different, they are considered two separate sites, so there’s a good chance to get penalized for duplicate content. If the search engine discovers two identical pages, generally it would take the page it saw first and ignore the other pages.The solutions are described below:

S__olutions:

1. Be smart about the site structure:  to keep the engines from crawling and indexing HTTPS pages, structure the website so that HTTPs are only accessible through a form submission (log-in, sign-up, or payment pages). The common mistake is making these pages available via a standard link (happens when you are either ignorant or  not aware that the secure version of the site is being crawled and indexed).
2. Use Robots.txt file to control which pages will be crawled and indexed
3. Use.htaccess file. Here’s how to do this:
4. Create a file names robots_ssl.txt in your root.
5. Add the following code to your .htaccessRewriteCond %{SERVER_PORT} 443 [NC]RewriteRule ^robots.txt$ robots_ssl.txt [L]
6. Remove yourdomain.com:443 from the webmaster tools if the pages have already been crawled
7. For dynamic pages like php, try< ?phpif ($_SERVER["SERVER_PORT"] == 443){echo “< meta name=” robots ” content=” noindex,nofollow ” > “;}?>
8. Dramatic solution (may not always be possible): 301 redirect the HTTPS pages to the HTTP pages – with hopes that the link juice will transfer over.

For more information please refer to this link :

http://www.seomoz.org/ugc/solving-duplicate-content-issues-with-http-and-https

I'm sure that your problem is solved.

[Kotkov]

Harald, " robots_ssl.txt " where did you get that?

[GKLA]

I would check your server for a https folder.

add a robots.txt file in the root of the https folder:

User-agent: *
Disallow:/

My guess is that the spider is following a link somewhere within your site that links to a https:// url.  The spider is than re-indexing the entire site using https://

My 2 cents for what its worth.

[mediabase]

Hi Serge, I came to know about the "robots_ssl.txt" from the website  http://www.seoworkers.com/seo-articles-tutorials/robots-and-https.html

[Dr-Pete]

I think Harald and James covered the bases here, but a couple of comments on Harald's reply:

(1) Definitely check this. A common cause of indexed https: pages is that a secure section of your site is being crawled (like a shopping cart), and you're using relative navigation links (like ) - when a crawler or visitor hits the nav link from a secure page, the relative link grabs the https: In most cases, you may want to NOINDEX secure pages. Shopping carts and checkout pages have no business in the search index, IMO.

[(2)-(5) I believe this does work, but it's very tricky, so please be careful. If anyone has linked to the https: pages, you'll lose the link-juice this way (you'll just cut those pages off). I honestly don't think it's a good choice for most sites.

(8) I actually believe the 301-redirect is simpler in most cases.

As James said, sitewide canonical tags (or on the affect pages, if they're isolated) will also work.](/contact.php)

[hawkvt1]

I would personally like to thank everyone that responded with an answer.  Man O Man, the best part of belonging to SEOMOZ is the community forum.  It's incredibly valuable, being able to ask a question and reach out to such talent as all of you.

If anyone ever gets up to Killington or Okemo skiing, the beer is on me!  I live right between both ski areas, about 8 miles to either mountain..

Thanks again.

[hawkvt1]

Thanks James..

Sorry, I was using Big Company as an example and just being generic.

The real URL if interested is www.hawkresort.com

[randfish]

Thanks dude! If I make it to Vermont, I might look you up

[hawkvt1]

Anytime Rand!  I only have two simple rules:

1.  Talking business on ski days is not allowed

2.  Entry into Vermont requires a pound of Seattle's best french roast coffee.  In return, you  receive some fantastic Vermont maple syrup.

Simple rules to live by LOL

Thanks again for all of your help...

Peter

[GTGshops]

Hi,

I'm still having problems with redirecting. I only have 1 duplicate page with https and http, that I want to redirect but it's the homepage.

i want to redirect: https://www.domain.com to http://www.domain.com

But keep the rest of the pages the same (half http and the other half https).

How do i do this?

[ajiabs]

Hate to respond to a 3 year old thread. But does this solution needs to be updated?

Is there any change in response now, as Google is favoring https for most pages. Does google still consider http and https as two different sites? If so which one should be suppressed - http or https?

Aji

[Dr-Pete]

If Google detects both http: and https: versions, they've started to automatically pick the https: version, but that's not consistent yet. In general, I think it's still important to set strong canonicalization signals. Google still separates your http: and https: sites in Google Search Console, too, so even they haven't quite made up their minds.

In general, Google is pushing sites toward https:, but that's a somewhat complex decision that depends on more than just SEO. If you're using https: and the https: URLs are indexed, then you should treat those as canonical and suppress the http: URLs, in most cases.

[GrowthLedge]

I'm reading this response and this is happening on my site as well.  How did this happen in the first place?  I have duplicate content because of https and http copies of all my web pages.  If I type https://www.mywebsite.com I can't get to my site.  Could this be coming from my hosting company?  I've set up my site to simply be http://www.mywebsite.com.  I'm a little worried to change my robots.txt and I would love to know how this happened in the first place.

[Dr-Pete]

Hard to tell without knowing the site, but it's possible there are external links to "https" versions of the pages. At this point, Google is going to increase the pressure to secure sites, and later this year Chrome will start warning users about all non-secure pages, so it may be worth making the move.