The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Robots.txt advice

    Robots.txt advice

    Intermediate & Advanced SEO
    3 3 105
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • eLab_London
      eLab_London last edited by

      Hey Guys,

      Have you ever seen coding like this in a robots.txt, I have never seen a noindex rule in a robots.txt file before - have you?

      user-agent: AhrefsBot
      

      User-agent: trovitBot
      User-agent: Nutch
      User-agent: Baiduspider
      Disallow: /

      User-agent: *
      Disallow: /WebServices/
      Disallow: /*?notfound=
      Disallow: /
      ?list=
      Noindex: /
      ?*list=
      Noindex: /local/
      Disallow: /local/
      Noindex: /handle/
      Disallow: /handle/
      Noindex: /Handle/
      Disallow: /Handle/
      Noindex: /localsites/
      Disallow: /localsites/
      Noindex: /search/
      Disallow: /search/
      Noindex: /Search/
      Disallow: /Search/
      Disallow: ?

      I have never seen a noindex rule in a robots.txt file before - have you?
      Any pointers?

      1 Reply Last reply Reply Quote 0
      • Tylerj
        Tylerj last edited by

        Best I could find was-

        Unlike disallowed pages, noindexed pages don’t end up in the index and therefore won’t show in search results. Combine both in robots.txt to optimise your crawl efficiency: the noindex will stop the page showing in search results, and the disallow will stop it being crawled

        From-https://www.deepcrawl.com/blog/best-practice/robots-txt-noindex-the-best-kept-secret-in-seo/

        1 Reply Last reply Reply Quote 0
        • Martijn_Scheijbeler
          Martijn_Scheijbeler last edited by

          Never seen this, doubt it's any useful as this isn't part of any search engines recommended statements to use. I don't think this would have any impact on what search engine robots would look at as it's not a statement in the robots.txt documentation.

          1 Reply Last reply Reply Quote 1
          • 1 / 1
          • First post
            Last post
          • Is robots met tag a more reliable than robots.txt at preventing indexing by Google?
            Bobbi_Tschumper
            Bobbi_Tschumper
            1
            7
            3.0k

          • Twitter Robots.TXT
            MarketingChimp10
            MarketingChimp10
            0
            5
            453

          • Robots.txt Allowed
            GlobeRunner
            GlobeRunner
            0
            4
            118

          • Have a Robots.txt Issue
            MattRoney
            MattRoney
            0
            5
            226

          • Meta robots or robot.txt file?
            Andy.Drinkwater
            Andy.Drinkwater
            0
            5
            152

          • Robots.txt assistance
            theLotter
            theLotter
            0
            9
            280

          • Robots.txt Syntax
            MichaelC-15022
            MichaelC-15022
            0
            2
            118

          • Robots.txt Question
            Dr-Pete
            Dr-Pete
            0
            4
            231

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy