The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Can I use a "no index, follow" command in a robot.txt file for a certain parameter on a domain?

    Can I use a "no index, follow" command in a robot.txt file for a certain parameter on a domain?

    Intermediate & Advanced SEO
    3 2 460
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • PapaRelevance
      PapaRelevance last edited by

      I have a site that produces thousands of pages via file uploads. These pages are then linked to by users for others to download what they have uploaded.

      Naturally, the client has blocked the parameter which precedes these pages in an attempt to keep them from being indexed. What they did not consider, was they these pages are attracting hundreds of thousands of links that are not passing any authority to the main domain because they're being blocked in robots.txt

      Can I allow google to follow, but NOT index these pages via a robots.txt file --- or would this have to be done on a page by page basis?

      1 Reply Last reply Reply Quote 0
      • PapaRelevance
        PapaRelevance last edited by

        Answered my own question.

        watch?v=ZjRGkc__FwQ

        NakulGoyal 1 Reply Last reply Reply Quote 0
        • NakulGoyal
          NakulGoyal @PapaRelevance last edited by

          Since you have those pages blocked via robots.txt, the bots would never even crawl these pages in theory...which means the Noindex,follow is not helping.

          Also, if you do a report on the domain on opensiteexplorer and dig, you should be able to find tons of those links already showing up. So if my site is linking to a page on that site, that page may not be cached/indexed because of the robots.txt exclusion, but that as long as my site is follow, your domain is still getting the credit for the link.

          Does that make sense ?

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Is robots met tag a more reliable than robots.txt at preventing indexing by Google?
            Bobbi_Tschumper
            Bobbi_Tschumper
            1
            7
            3.0k

          • If Robots.txt have blocked an Image (Image URL) but the other page which can be indexed has this image, how is the image treated?
            alphonseha
            alphonseha
            1
            4
            1.4k

          • Meta robots or robot.txt file?
            Andy.Drinkwater
            Andy.Drinkwater
            0
            5
            152

          • Dilemma about "images" folder in robots.txt
            dancape
            dancape
            1
            3
            3.4k

          • How important is a good "follow" / "no-follow" link ratio for SEO?
            TakeshiYoung
            TakeshiYoung
            0
            3
            1.4k

          • SEO & Magento Multistore - I have been asked if "duplicatiing" a magento stor using its "Multistore" functionality will cause both to be picked up as duplicate content, can anybody help?
            SanketPatel
            SanketPatel
            0
            2
            376

          • Using 2 wildcards in the robots.txt file
            lonniea
            lonniea
            0
            2
            605

          • How Can I know that a link placed is not lableld "No Follow"?
            ChristopherGlaeser
            ChristopherGlaeser
            0
            4
            526

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy