The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. On-Page / Site Optimization
    4. Disallow indexing of ALL subdomains

    Disallow indexing of ALL subdomains

    On-Page / Site Optimization
    5 3 178
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • conversal
      conversal last edited by

      I'm using www.domain.com as my development hosting. Each website that i'm developing get's a temporary URL like this:

      project1.domain.com
      project2.domain.com
      project3.domain.com
      ...

      Now i'd like to set that ALL these subdomains can not be indexed in Google. Now I manually have to do this for each subdomain's site, and when I go online I have to change the robots.txt again. So I would like to make things a bit easier for me.

      Is this possible?

      1 Reply Last reply Reply Quote 0
      • WesleySmits
        WesleySmits last edited by

        You should edit your .htaccess file as is described here:
        http://stackoverflow.com/questions/6738896/excluding-testing-subdomain-from-being-crawled-by-search-engines-w-svn-reposit

        I hope i answered your question 🙂

        1 Reply Last reply Reply Quote 0
        • conversal
          conversal last edited by

          Thanks for the link, but I don't think that will solve my problem?

          When Google wants to crawl project1.domain.com, he won't check the .htaccess of domain.com right? So then he'll just be crawling that subdomain?

          1 Reply Last reply Reply Quote 0
          • conversal
            conversal last edited by

            Nobody?

            1 Reply Last reply Reply Quote 0
            • Aleyda
              Aleyda last edited by

              Hello there!

              Like Wesley mentioned before the best way to avoid any "non-desired" crawling and indexing of your development or testing environment is by requiring authentication (whether with htaccess or your own programmed login/password screen).

              Unfortunately sometimes there have been situations when search crawlers don't necessarily follow robots.txt directives. Additionally, beyond search engines you might want to protect your development or testing environment, making sure that only people with the required access to it can enter. Because of this, the best way to go is requiring authentication to access to your subdomains, not robots.txt.

              I hope this help!

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              • First post
                Last post
              • Website Not Indexing
                varun1800
                varun1800
                0
                3
                233

              • Index or No Index (Panda Issue)
                Carson-Ward
                Carson-Ward
                0
                5
                371

              • Blocking Subdomain from Google Crawl and Index
                odmsoft
                odmsoft
                0
                8
                33.7k

              • Google Indexed = 35, 445 pages, Bing Indexed = 243 pages... Why?
                zDucketz
                zDucketz
                0
                7
                479

              • When to not index
                irvingw
                irvingw
                0
                3
                135

              • Domain and subdomain comparison - a deeper look into the subdomain metrics
                Cyrus-Shepard
                Cyrus-Shepard
                0
                7
                927

              • Why some pages are not indexed?
                Conor_OShea_ETUS
                Conor_OShea_ETUS
                0
                2
                641

              • Page Indexing
                IM_Learner
                IM_Learner
                0
                2
                868

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy