The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Robots txt. in page with 301 redirect

    Robots txt. in page with 301 redirect

    Technical SEO Issues
    6 2 3.1k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Kilgray
      Kilgray last edited by

      We currently have a a series of help pages that we would like to disallow from our robots txt.

      The thing is that these help pages are located in our old website, which now has a 301 redirect to current site.

      Which is the proper way to go around?

      1- Add the pages we want to disallow to the robots.txt of the new website?

      2- Break the redirect momentarily and add the pages to the robots.txt of the old one?

      Thanks

      1 Reply Last reply Reply Quote 0
      • LauraSultan
        LauraSultan last edited by

        Do you want to disallow the URLs that these pages are being redirected to? If not, there's no need to add anything to the robots.txt file.

        If you do want to disallow the URLs that these pages are being redirected to, use relative URLs in your robots.txt file. For example, let's say olddomain.com/old-help-page/ is being redirected to newdomain.com/new-help-page/. If that's the case, add the following to your robots.txt file.

        Disallow: /new-help-page/

        There's no need to disallow the specific URLs that are being redirected to something else.  Are you trying to get them removed from Google's index or something?  If so, Google will update their index eventually based on your 301 redirects.

        Kilgray 1 Reply Last reply Reply Quote 0
        • Kilgray
          Kilgray @LauraSultan last edited by

          Hi Laura,

          Thanks for your reply. I don't want to disallow the URLs these pages are being redirected to. Actually these URLs are in the old version but still can be accessed. So to put it simply, this is my case:

          1- This was our current website: www.kilgray.com (With a 301 redirect)

          2- This is our new website: www.memoq.com

          3- I would like to disallow the following links on the old website that are still visible (haven't been redirected):

          http://kilgray.com/memoq/2015-100/help-en/index.html

          http://kilgray.com/memoq/2014/help-en/

          http://kilgray.com/memoq/2013R2/help-full-en/

          http://kilgray.com/memoq/60/help-en/

          LauraSultan 1 Reply Last reply Reply Quote 0
          • LauraSultan
            LauraSultan @Kilgray last edited by

            So, the problem is that the robots.txt file can't be accessed because of the 301 redirect to the new domain?

            Do you plan to keep the help files on the old domain, or will they be removed completely?

            Kilgray 1 Reply Last reply Reply Quote 0
            • Kilgray
              Kilgray @LauraSultan last edited by

              Hey, for some time we will keep the files in the old domain. Should we break the redirect and insert the disallows to the robot.txt of the old site?

              LauraSultan 1 Reply Last reply Reply Quote 0
              • LauraSultan
                LauraSultan @Kilgray last edited by

                In that case, you'd need to add the robots meta tag at the page level before the tag.

                or

                1 Reply Last reply Reply Quote 1
                • 1 / 1
                • First post
                  Last post
                • Page for page 301 redirects from old server to new server
                  cindyt-17038
                  cindyt-17038
                  0
                  5
                  398

                • 301 redirects - one overall redirect or an individual one for each page url
                  Kurt_Steinbrueck
                  Kurt_Steinbrueck
                  1
                  4
                  160

                • Redirect a 301 Redirect
                  RedCaffeine
                  RedCaffeine
                  0
                  5
                  98

                • Site blocked by robots.txt and 301 redirected still in SERPs
                  OlegKorneitchouk
                  OlegKorneitchouk
                  0
                  2
                  621

                • How do I 301 redirect a number of pages to one page
                  AndersS
                  AndersS
                  0
                  5
                  404

                • Is it better to delete web pages that I don't want anymore or should I 301 redirect all of the pages I delete to the homepage or another live page?
                  EGOL
                  EGOL
                  0
                  2
                  417

                • 301 Redirect Domain or 301 Redirect Domain + Interior Pages
                  Room214
                  Room214
                  0
                  7
                  1.1k

                • 301 redirecting some pages directly, and the rest to a single page
                  danrawk
                  danrawk
                  0
                  3
                  703

                Get started with Moz Pro!

                Unlock the power of advanced SEO tools and data-driven insights.

                Start my free trial
                Products
                • Moz Pro
                • Moz Local
                • Moz API
                • Moz Data
                • STAT
                • Product Updates
                Moz Solutions
                • SMB Solutions
                • Agency Solutions
                • Enterprise Solutions
                • Digital Marketers
                Free SEO Tools
                • Domain Authority Checker
                • Link Explorer
                • Keyword Explorer
                • Competitive Research
                • Brand Authority Checker
                • Local Citation Checker
                • MozBar Extension
                • MozCast
                Resources
                • Blog
                • SEO Learning Center
                • Help Hub
                • Beginner's Guide to SEO
                • How-to Guides
                • Moz Academy
                • API Docs
                About Moz
                • About
                • Team
                • Careers
                • Contact
                Why Moz
                • Case Studies
                • Testimonials
                Get Involved
                • Become an Affiliate
                • MozCon
                • Webinars
                • Practical Marketer Series
                • MozPod
                Connect with us

                Contact the Help team

                Join our newsletter
                Moz logo
                © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                • Accessibility
                • Terms of Use
                • Privacy