The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. How to block google robots from a subdomain

    How to block google robots from a subdomain

    Technical SEO Issues
    5 2 2.6k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Alexey_mindvalley
      Alexey_mindvalley last edited by

      I have a subdomain that lets me preview the changes I put on my site.

      The live site URL is www.site.com, working preview version is www.site.edit.com

      The contents on both are almost identical

      I want to block the preview version (www.site.edit.com) from Google Robots, so that they don't penalize me for duplicated content.

      Is it the right way to do it:

      User-Agent: *

      Disallow: .edit.com/*

      1 Reply Last reply Reply Quote 0
      • Martijn_Scheijbeler
        Martijn_Scheijbeler last edited by

        Hi,

        The Google Robots will look for the robots.txt in each individual root. So you need the robots.txt in the root of the subdomain not just the domain root. That's why its also possible to include a complete disallow in there and not just: .edit.com/* .

        Example:
        User-agent: *
        Disallow: /

        Hope this helps!

        Alexey_mindvalley 1 Reply Last reply Reply Quote 1
        • Alexey_mindvalley
          Alexey_mindvalley @Martijn_Scheijbeler last edited by

          Thanks a lot for your answer, Martijn!

          So just to make sure I got it correctly - this robots file URL should be:

          www.site.edit.com/robots.txt

          ?

          Thanks a lot for your answer

          Martijn_Scheijbeler 1 Reply Last reply Reply Quote 0
          • Martijn_Scheijbeler
            Martijn_Scheijbeler @Alexey_mindvalley last edited by

            Hi,

            Probably without the www. so: site.edit.com/robots.txt because otherwise you would have a subdomain of a subdomain ;-). But the rest is perfect!

            1 Reply Last reply Reply Quote 1
            • Alexey_mindvalley
              Alexey_mindvalley last edited by

              Thanks o much for your help!

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              • First post
                Last post
              • Google Webmaster Tools is saying "Sitemap contains urls which are blocked by robots.txt" after Https move...
                vetofunk
                vetofunk
                0
                5
                11.2k

              • Google is Still Blocking Pages Unblocked 1 Month ago in Robots
                Tyler123
                Tyler123
                0
                3
                131

              • Blocking subdomains without blocking sites...
                OlegKorneitchouk
                OlegKorneitchouk
                1
                6
                145

              • Google index dymamic webpages after block in robots.txt...
                Dr-Pete
                Dr-Pete
                0
                6
                247

              • Google insists robots.txt is blocking... but it isn't.
                Christy-Correll
                Christy-Correll
                0
                4
                523

              • Blocked by robots
                ThompsonPaul
                ThompsonPaul
                0
                7
                211

              • Google (GWT) says my homepage and posts are blocked by Robots.txt
                stubby
                stubby
                0
                8
                930

              • Warnings for blocked by blocked by meta-robots/meta robots Nofollow...how to resolve?
                Cyrus-Shepard
                Cyrus-Shepard
                0
                3
                415

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy