The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Robots.txt on http vs. https

    Robots.txt on http vs. https

    Technical SEO Issues
    4 2 7.0k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • zeepartner
      zeepartner last edited by

      We recently changed our domain from http to https. When a user enters any URL on http, there is an global 301 redirect to the same page on https.

      I cannot find instructions about what to do with robots.txt. Now that https is the canonical version, should I block the http-Version with robots.txt?

      Strangely, I cannot find a single ressource about this...

      1 Reply Last reply Reply Quote 0
      • BlueprintMarketing
        BlueprintMarketing last edited by

        "Now that https is the canonical version, should I block the http-Version with robots.txt?"

        Absolutely not GWT will handel all of it think about backlinks both https:// & http:// urls you will not want to lose the flow of link juice that you would cut off

        Remake robost.txt with

        http://www.internetmarketingninjas.com/seo-tools/robots-txt-generator/

        But use https:// for the xml sitemap.

        1 Reply Last reply Reply Quote 2
        • zeepartner
          zeepartner last edited by

          Good point with the backlinks! Currently, both robots.txt files are open and google does not seem to have canonicalization problems so far. So it makes sense to leave it this way anyways... Thanks Thomas!

          1 Reply Last reply Reply Quote 0
          • BlueprintMarketing
            BlueprintMarketing last edited by

            Glad to be of help. Check out this Google link to confirm you picked up the 180 day crawl

            https://support.google.com/webmasters/answer/83106?hl=en

            Second URLs helpful as well.

            http://blog.raventools.com/moving-site-from-http-to-ssl/

            all the best,

            tom

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • Http:// vs Https:// in Og:URL
              ThompsonPaul
              ThompsonPaul
              0
              6
              637

            • #1 rankings on both HTTP and HTTPS vs duplicate content
              dohertyjf
              dohertyjf
              0
              3
              110

            • Robots.txt vs. meta noindex, follow
              Andrew_Birkitt
              Andrew_Birkitt
              0
              3
              1.1k

            • Google Search Console Site Map Anomalies (HTTP vs HTTPS)
              Dan-Lawrence
              Dan-Lawrence
              0
              3
              1.0k

            • Http vs https: which is better for seo / usability
              kwoolf
              kwoolf
              0
              2
              671

            • Robots.txt Download vs Cache
              FedeEinhorn
              FedeEinhorn
              0
              2
              128

            • Http VS https and google crawl and indexing ?
              sherohass
              sherohass
              0
              4
              1.8k

            • I have a site that has both http:// and https:// versions indexed, e.g. https://www.homepage.com/ and http://www.homepage.com/. How do I de-index the https// versions without losing the link juice that is going to the https://homepage.com/ pages?
              fthead9
              fthead9
              0
              3
              729

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy