The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Robots.txt help

    Robots.txt help

    Intermediate & Advanced SEO
    4 3 85
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • IceIcebaby
      IceIcebaby last edited by

      Hi Moz Community,

      Google is indexing some developer pages from a previous website where I currently work:

      ddcblog.dev.examplewebsite.com/categories/sub-categories

      Was wondering how I include these in a robots.txt file so they no longer appear on Google. Can I do it under our homepage GWT account or do I have to have a separate account set up for these URL types?

      As always, your expertise is greatly appreciated,

      -Reed

      1 Reply Last reply Reply Quote 0
      • KeriMorgret
        KeriMorgret last edited by

        You'll want a separate account for that subdomain, and also put the robots.txt excluding that subdomain in that subdomain itself.

        1 Reply Last reply Reply Quote 1
        • LesleyPaone
          LesleyPaone last edited by

          Can you not just add a htaccess password to the directory to keep the dev site up, but keep bots out?

          KeriMorgret 1 Reply Last reply Reply Quote 1
          • KeriMorgret
            KeriMorgret @LesleyPaone last edited by

            The robots.txt would allow the OP to go back into GWT and request removal of the dev site from the index. Password protecting a dev site is usually a pretty good idea, too.

            1 Reply Last reply Reply Quote 1
            • 1 / 1
            • First post
              Last post
            • Robots.txt Help
              GlobeRunner
              GlobeRunner
              0
              5
              162

            • Need help with Robots.txt
              MattAntonino
              MattAntonino
              1
              3
              127

            • Robots.txt question
              Mobilio
              Mobilio
              0
              2
              107

            • Have a Robots.txt Issue
              MattRoney
              MattRoney
              0
              5
              226

            • Avoiding Duplicate Content with Used Car Listings Database: Robots.txt vs Noindex vs Hash URLs (Help!)
              Everett
              Everett
              0
              10
              2.0k

            • Robot.txt help
              evolvingSEO
              evolvingSEO
              0
              23
              203

            • Robot.txt error
              Rubix
              Rubix
              0
              13
              203

            • Robots.txt Question
              Dr-Pete
              Dr-Pete
              0
              4
              231

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy