The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. On-Page / Site Optimization
    4. Disallow indexing of ALL subdomains

    Disallow indexing of ALL subdomains

    On-Page / Site Optimization
    5 3 178
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • conversal
      conversal last edited by

      I'm using www.domain.com as my development hosting. Each website that i'm developing get's a temporary URL like this:

      project1.domain.com
      project2.domain.com
      project3.domain.com
      ...

      Now i'd like to set that ALL these subdomains can not be indexed in Google. Now I manually have to do this for each subdomain's site, and when I go online I have to change the robots.txt again. So I would like to make things a bit easier for me.

      Is this possible?

      1 Reply Last reply Reply Quote 0
      • WesleySmits
        WesleySmits last edited by

        You should edit your .htaccess file as is described here:
        http://stackoverflow.com/questions/6738896/excluding-testing-subdomain-from-being-crawled-by-search-engines-w-svn-reposit

        I hope i answered your question 🙂

        1 Reply Last reply Reply Quote 0
        • conversal
          conversal last edited by

          Thanks for the link, but I don't think that will solve my problem?

          When Google wants to crawl project1.domain.com, he won't check the .htaccess of domain.com right? So then he'll just be crawling that subdomain?

          1 Reply Last reply Reply Quote 0
          • conversal
            conversal last edited by

            Nobody?

            1 Reply Last reply Reply Quote 0
            • Aleyda
              Aleyda last edited by

              Hello there!

              Like Wesley mentioned before the best way to avoid any "non-desired" crawling and indexing of your development or testing environment is by requiring authentication (whether with htaccess or your own programmed login/password screen).

              Unfortunately sometimes there have been situations when search crawlers don't necessarily follow robots.txt directives. Additionally, beyond search engines you might want to protect your development or testing environment, making sure that only people with the required access to it can enter. Because of this, the best way to go is requiring authentication to access to your subdomains, not robots.txt.

              I hope this help!

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              • First post
                Last post
              • Index or No Index (Panda Issue)
                Carson-Ward
                Carson-Ward
                0
                5
                371

              • Does this index well
                neilhenderson
                neilhenderson
                0
                4
                113

              • Subdomain for languages
                David-Kley
                David-Kley
                0
                9
                116

              • Indexing pages after de-indexing them
                MohammadSabbagh
                MohammadSabbagh
                0
                4
                96

              • No index, or no index no follow?
                AuthorityLinkPartners
                AuthorityLinkPartners
                0
                3
                700

              • When to not index
                irvingw
                irvingw
                0
                3
                135

              • How do I do a 301 Redirect in IIS 7 from http://www.freightmonster.com/index.html to http://freightmonster.com/index.html when I don't have a physical page to redirect?
                EricaMcGillivray
                EricaMcGillivray
                0
                4
                1.0k

              • No Index help
                Vetshopgroup
                Vetshopgroup
                0
                5
                658

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy