The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Best method for blocking a subdomain with duplicated content

    Best method for blocking a subdomain with duplicated content

    Intermediate & Advanced SEO
    5 4 97
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • KateWaite
      KateWaite last edited by

      Hello Moz Community

      Hoping somebody can assist.

      We have a subdomain, used by our CMS, which is being indexed by Google. 
      http://www.naturalworldsafaris.com/
      https://admin.naturalworldsafaris.com/

      The page is the same so we can't add a no-index or no-follow. 
      I have both set up as separate properties in webmaster tools

      I understand the best method would be to update the robots.txt with a user disallow for the subdomain - but the robots text is only accessible on the main domain. http://www.naturalworldsafaris.com/robots.txt Will this work if we add the subdomain exclusion to this file?

      It means it won't be accessible on https://admin.naturalworldsafaris.com/robots.txt (where we can't create a file). Therefore won't be seen within that specific webmaster tools property.

      I've also asked the developer to add a password protection to the subdomain but this does not look possible.

      What approach would you recommend?

      1 Reply Last reply Reply Quote 0
      • RyanPurkey
        RyanPurkey last edited by

        That's very strange behavior for the admin portion of a site to not be behind password protection.  If the current developer is unable to do so, I'd ask around on that because if you block the admin portion via password protection, even if it's still indexed in Google it will rank much lower.

        Are you able to contact someone at CMS Source to get some support?  That might help resolve this as well as provide guidance on getting the robots.txt uploaded and being able to add noindex to admin only.

        1 Reply Last reply Reply Quote 2
        • MibuKotaro
          MibuKotaro last edited by

          One alternative you could try is to place a while this is not as ideal as adding the noindex nofollow it could help tell Google that the admin is not the preferred domain.

          1 Reply Last reply Reply Quote 1
          • LynnPatchett
            LynnPatchett last edited by

            HI,

            If you have the subdomain set up in GWT then you could do a url removal request (Google Index -> Remove URLs). Just put in the root admin subdomain (ie https://admin.naturalworldsafaris.com/). This usually works pretty quickly and given your description will likely be the quickest way to drop admin urls that are showing up in the serps.

            It is not an ideal solution and the tool will tell you that for permanent exclusion you should do it through your robots.txt file. in regards your question about that though - the robots.txt file for the subdomain has to be placed on the subdomain, adding the subdomain to the main domains robot.txt file wont work.

            1 Reply Last reply Reply Quote 1
            • KateWaite
              KateWaite last edited by

              Many thanks for taking the time to reply.

              Ryan - I am talking to CMS Source about these options; it's not possible with the current configuration but it looks like we may be able to change this with a bit of development work.

              Sean - Thanks, I had considered the canonical, which would at least prevent a duplicate content issue although I think we need to take measures to stop this admin. subdomain being accessible which this won't help with.

              Lynn - Thanks, I'll do that as an interim measure.

              Thanks
              Kate

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              • First post
                Last post
              • How to find duplicate content, boilerplate content (repeated content) for entire website?
                Alick300
                Alick300
                0
                3
                110

              • Duplicate content within sections of a page but not full page duplicate content
                J_Sinclair
                J_Sinclair
                0
                3
                112

              • Best tools for identifying internal duplicate content
                Churchill1
                Churchill1
                0
                9
                32.1k

              • Best way to remove duplicate content with categories?
                NakulGoyal
                NakulGoyal
                0
                3
                165

              • How to Best Establish Ownership when Content is Duplicated?
                Allie_Williams
                Allie_Williams
                0
                5
                257

              • How best to handle (legitimate) duplicate content?
                Aran_Smithson
                Aran_Smithson
                0
                13
                856

              • Duplicate Content On A Subdomain
                irvingw
                irvingw
                0
                2
                685

              • Subdomains - duplicate content - robots.txt
                SeoStallion
                SeoStallion
                0
                4
                890

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy