The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Robots.txt Disallowed Pages and Still Indexed

    Robots.txt Disallowed Pages and Still Indexed

    Intermediate & Advanced SEO
    3 3 2.9k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DRSearchEngOpt
      DRSearchEngOpt last edited by

      Alright, I am pretty sure I know the answer is "Nothing more I can do here." but I just wanted to double check.  It relates to the robots.txt file and that pesky "A description for this result is not available because of this site's robots.txt". Typically people want the URL indexed and the normal Meta Description to be displayed but I don't want the link there at all. I purposefully am trying to robots that stuff outta there.
      My question is, has anybody tried to get a page taken out of the Index and had this happen; URL still there but pesky robots.txt message for meta description?  Were you able to get the URL to no longer show up or did you just live with this?

      Thanks folks, you are always great!

      1 Reply Last reply Reply Quote 0
      • LoganRay
        LoganRay last edited by

        Hi Chris,

        This is a common misconception about the robots.txt file. It's not intended to remove URLs from the index, only to prevent crawling of URLs/folders/parameters/etc.

        Here's a quote straight from the Search Console Help site: "You should not use robots.txt as a means to hide your web pages from Google Search results."

        You can temporarily remove that URL using Search Console. To permanently remove it, you'll need to add a meta robots noindex tag, the only sure-fire way to prevent indexation. Here's what that tag looks like:

        Igor.Go 1 Reply Last reply Reply Quote 3
        • Igor.Go
          Igor.Go @LoganRay last edited by

          And don't forget to remove disallow in robots.txt first, if you want to remove it from index. Because if you add meta nofollow while the page is disallowed it won't go anywhere, crawler will not check it and it will stay indexed.

          Allow > Add meta noindex > wait for it to be deindexed > Disallow

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Robots.txt, Disallow & Indexed-Pages..
            thekiller99
            thekiller99
            0
            5
            341

          • Will disallowing URL's in the robots.txt file stop those URL's being indexed by Google
            Martijn_Scheijbeler
            Martijn_Scheijbeler
            0
            11
            1.6k

          • Pages getting into Google Index, blocked by Robots.txt??
            Devanur-Rafi
            Devanur-Rafi
            0
            10
            673

          • If i disallow unfriendly URL via robots.txt, will its friendly counterpart still be indexed?
            Cyrus-Shepard
            Cyrus-Shepard
            0
            4
            620

          • How to Disallow Tag Pages With Robot.txt
            monster99
            monster99
            0
            6
            4.0k

          • Do I need to disallow the dynamic pages in robots.txt?
            esiow2013
            esiow2013
            0
            11
            1.0k

          • Old pages still crawled by SE returning 404s. Better to put 301 or block with robots.txt ?
            RyanKent
            RyanKent
            0
            2
            721

          • Category Pages - Canonical, Robots.txt, Changing Page Attributes
            Function5
            Function5
            0
            9
            1.1k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy