The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Robots.txt help

    Robots.txt help

    Intermediate & Advanced SEO
    4 3 85
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • IceIcebaby
      IceIcebaby last edited by

      Hi Moz Community,

      Google is indexing some developer pages from a previous website where I currently work:

      ddcblog.dev.examplewebsite.com/categories/sub-categories

      Was wondering how I include these in a robots.txt file so they no longer appear on Google. Can I do it under our homepage GWT account or do I have to have a separate account set up for these URL types?

      As always, your expertise is greatly appreciated,

      -Reed

      1 Reply Last reply Reply Quote 0
      • KeriMorgret
        KeriMorgret last edited by

        You'll want a separate account for that subdomain, and also put the robots.txt excluding that subdomain in that subdomain itself.

        1 Reply Last reply Reply Quote 1
        • LesleyPaone
          LesleyPaone last edited by

          Can you not just add a htaccess password to the directory to keep the dev site up, but keep bots out?

          KeriMorgret 1 Reply Last reply Reply Quote 1
          • KeriMorgret
            KeriMorgret @LesleyPaone last edited by

            The robots.txt would allow the OP to go back into GWT and request removal of the dev site from the index. Password protecting a dev site is usually a pretty good idea, too.

            1 Reply Last reply Reply Quote 1
            • 1 / 1
            • First post
              Last post
            • Twitter Robots.TXT
              MarketingChimp10
              MarketingChimp10
              0
              5
              453

            • Robots.txt Allowed
              GlobeRunner
              GlobeRunner
              0
              4
              118

            • Meta robots or robot.txt file?
              Andy.Drinkwater
              Andy.Drinkwater
              0
              5
              152

            • Help with Robots.txt On a Shared Root
              donford
              donford
              0
              10
              121

            • Robots.txt
              Travis_Bailey
              Travis_Bailey
              0
              4
              107

            • Robots.txt Syntax
              MichaelC-15022
              MichaelC-15022
              0
              2
              118

            • Robot.txt help
              evolvingSEO
              evolvingSEO
              0
              23
              203

            • Robots.txt unblock
              Elchanan
              Elchanan
              0
              5
              4.3k

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy