The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Robots txt. in page with 301 redirect

    Robots txt. in page with 301 redirect

    Technical SEO Issues
    6 2 3.1k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Kilgray
      Kilgray last edited by

      We currently have a a series of help pages that we would like to disallow from our robots txt.

      The thing is that these help pages are located in our old website, which now has a 301 redirect to current site.

      Which is the proper way to go around?

      1- Add the pages we want to disallow to the robots.txt of the new website?

      2- Break the redirect momentarily and add the pages to the robots.txt of the old one?

      Thanks

      1 Reply Last reply Reply Quote 0
      • LauraSultan
        LauraSultan last edited by

        Do you want to disallow the URLs that these pages are being redirected to? If not, there's no need to add anything to the robots.txt file.

        If you do want to disallow the URLs that these pages are being redirected to, use relative URLs in your robots.txt file. For example, let's say olddomain.com/old-help-page/ is being redirected to newdomain.com/new-help-page/. If that's the case, add the following to your robots.txt file.

        Disallow: /new-help-page/

        There's no need to disallow the specific URLs that are being redirected to something else.  Are you trying to get them removed from Google's index or something?  If so, Google will update their index eventually based on your 301 redirects.

        Kilgray 1 Reply Last reply Reply Quote 0
        • Kilgray
          Kilgray @LauraSultan last edited by

          Hi Laura,

          Thanks for your reply. I don't want to disallow the URLs these pages are being redirected to. Actually these URLs are in the old version but still can be accessed. So to put it simply, this is my case:

          1- This was our current website: www.kilgray.com (With a 301 redirect)

          2- This is our new website: www.memoq.com

          3- I would like to disallow the following links on the old website that are still visible (haven't been redirected):

          http://kilgray.com/memoq/2015-100/help-en/index.html

          http://kilgray.com/memoq/2014/help-en/

          http://kilgray.com/memoq/2013R2/help-full-en/

          http://kilgray.com/memoq/60/help-en/

          LauraSultan 1 Reply Last reply Reply Quote 0
          • LauraSultan
            LauraSultan @Kilgray last edited by

            So, the problem is that the robots.txt file can't be accessed because of the 301 redirect to the new domain?

            Do you plan to keep the help files on the old domain, or will they be removed completely?

            Kilgray 1 Reply Last reply Reply Quote 0
            • Kilgray
              Kilgray @LauraSultan last edited by

              Hey, for some time we will keep the files in the old domain. Should we break the redirect and insert the disallows to the robot.txt of the old site?

              LauraSultan 1 Reply Last reply Reply Quote 0
              • LauraSultan
                LauraSultan @Kilgray last edited by

                In that case, you'd need to add the robots meta tag at the page level before the tag.

                or

                1 Reply Last reply Reply Quote 1
                • 1 / 1
                • First post
                  Last post
                • Will a Robots.txt 'disallow' of a directory, keep Google from seeing 301 redirects for pages/files within the directory?
                  DmitriiK
                  DmitriiK
                  0
                  4
                  408

                • Advice on whether we 301 redirect a page or update existing page?
                  tomwhite
                  tomwhite
                  0
                  2
                  83

                • Can you 301 redirect a page to an already existing/old page ?
                  Bryan_Loconto
                  Bryan_Loconto
                  0
                  3
                  262

                • Delete 301 redirected pages from server after redirect is in place?
                  RuthBurrReedy
                  RuthBurrReedy
                  0
                  5
                  4.0k

                • Robots.txt and 301
                  Bush_JSM
                  Bush_JSM
                  0
                  5
                  427

                • 301 redirecting some pages directly, and the rest to a single page
                  danrawk
                  danrawk
                  0
                  3
                  703

                • Robots.txt File Redirects to Home Page
                  NickPateman81
                  NickPateman81
                  0
                  6
                  4.1k

                • On Page 301 redirect for html pages
                  randfish
                  randfish
                  0
                  7
                  1.6k

                Get started with Moz Pro!

                Unlock the power of advanced SEO tools and data-driven insights.

                Start my free trial
                Products
                • Moz Pro
                • Moz Local
                • Moz API
                • Moz Data
                • STAT
                • Product Updates
                Moz Solutions
                • SMB Solutions
                • Agency Solutions
                • Enterprise Solutions
                • Digital Marketers
                Free SEO Tools
                • Domain Authority Checker
                • Link Explorer
                • Keyword Explorer
                • Competitive Research
                • Brand Authority Checker
                • Local Citation Checker
                • MozBar Extension
                • MozCast
                Resources
                • Blog
                • SEO Learning Center
                • Help Hub
                • Beginner's Guide to SEO
                • How-to Guides
                • Moz Academy
                • API Docs
                About Moz
                • About
                • Team
                • Careers
                • Contact
                Why Moz
                • Case Studies
                • Testimonials
                Get Involved
                • Become an Affiliate
                • MozCon
                • Webinars
                • Practical Marketer Series
                • MozPod
                Connect with us

                Contact the Help team

                Join our newsletter
                Moz logo
                © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                • Accessibility
                • Terms of Use
                • Privacy