The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Why is noindex more effective than robots.txt?

    Why is noindex more effective than robots.txt?

    Intermediate & Advanced SEO
    3 3 705
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • nicole.healthline
      nicole.healthline last edited by

      In this post, http://www.seomoz.org/blog/restricting-robot-access-for-improved-seo, it mentions that the noindex tag is more effective than using robots.txt for keeping URLs out of the index. Why is this?

      1 Reply Last reply Reply Quote 0
      • john4math
        john4math last edited by

        The disallow in robots.txt will prevent the bots from crawling that page, but will not prevent the page from appearing on SERPs.  If a page with a lot of links to it is disallowed in the robots.txt, it may still appear on SERPs.  I've seen this on a few of my own pages... and Google picks a weird title for the page...

        If you put the meta noindex tag, Google will actively remove that page from their search results when they re-index that page.

        Here was one webmaster central thread I found about it.

        webseoservices 1 Reply Last reply Reply Quote 2
        • webseoservices
          webseoservices @john4math last edited by

          Good answer, also we have seen that sometimes bots come directly to your site via a link and do not always visit the robots.txt file and will therefore index the first page they come to.

          Matt Cutts has said before that the only 100% fail safe way of blocking search engines indexing something will be to have it password protected 🙂

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Robots.txt
            Travis_Bailey
            Travis_Bailey
            0
            4
            107

          • "noindex, follow" or "robots.txt" for thin content pages
            khi5
            khi5
            0
            6
            1.5k

          • Robots.txt
            TomRayner
            TomRayner
            0
            5
            137

          • Robot.txt help
            evolvingSEO
            evolvingSEO
            0
            23
            203

          • Robot.txt error
            Rubix
            Rubix
            0
            13
            203

          • Panda Updates - robots.txt or noindex?
            dmccarthy
            dmccarthy
            0
            5
            666

          • Will disallowing in robots.txt noindex a page?
            FranckNlemba
            FranckNlemba
            0
            6
            510

          • Effect duration of robots.txt file.
            KeriMorgret
            KeriMorgret
            0
            4
            255

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy