The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Using folder blocked by robots.txt before uploaded to indexed folder - is that OK?

    Using folder blocked by robots.txt before uploaded to indexed folder - is that OK?

    Intermediate & Advanced SEO
    4 3 100
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • khi5
      khi5 last edited by

      I have a folder "testing" within my domain which is a folder added to the robots.txt. My web developers use that folder "testing" when we are creating new content before uploading to an indexed folder. So the content is uploaded to the "testing" folder at first (which is blocked by robots.txt) and later uploaded to an indexed folder, yet permanently keeping the content in the "testing" folder. Actually, my entire website's content is located within the "testing" - so same URL structure for all pages as indexed pages, except it starts with the "testing/" folder.

      Question: even though the "testing" folder will not be indexed by search engines, is there a chance search engines notice that the content is at first uploaded to the "testing" folder and therefore the indexed folder is not guaranteed to get the content credit, since search engines see the content in the "testing" folder, despite the "testing" folder being blocked by robots.txt? Would it be better that I password protecting this "testing" folder?

      Thx

      1 Reply Last reply Reply Quote 0
      • Ray-pp
        Ray-pp last edited by

        As long as the correct robots.txt setting has been applied to the /testing folder, then you do not have anything to worry about.

        Considering that it is a staging environment, I would recommend securing it with a password just to be safe and secure with your non-production site content.

        1 Reply Last reply Reply Quote 1
        • ecommercebc
          ecommercebc last edited by

          Yep, just to jump in on the above, if a competitor is paying attention to your robots.txt file, they might notice a sweet stash of content under the /testing folder that they can nab. I have actually seen something similar happen in the past in a competitive SEO niche, so something to bear in mind.

          khi5 1 Reply Last reply Reply Quote 2
          • khi5
            khi5 @ecommercebc last edited by

            good observation....

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • Is robots met tag a more reliable than robots.txt at preventing indexing by Google?
              Bobbi_Tschumper
              Bobbi_Tschumper
              1
              7
              3.0k

            • Robots.txt - Do I block Bots from crawling the non-www version if I use www.site.com ?
              PatrickDelehanty
              PatrickDelehanty
              0
              2
              479

            • Block in robots.txt instead of using canonical?
              RobertFisher
              RobertFisher
              0
              9
              1.6k

            • Google showing high volume of URLs blocked by robots.txt in in index-should we be concerned?
              TakeshiYoung
              TakeshiYoung
              0
              4
              302

            • What should I block with a robots.txt file?
              Travis-W
              Travis-W
              1
              3
              298

            • Why are new pages not being indexed, and old pages (now in robots.txt) remain in the index?
              KeriMorgret
              KeriMorgret
              0
              3
              378

            • What content should I block in wodpress with robots.txt?
              ENSO
              ENSO
              0
              4
              518

            • Block an entire subdomain with robots.txt?
              kylesuss
              kylesuss
              1
              16
              102.1k

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy