The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Robots.txt Disallowed Pages and Still Indexed

    Robots.txt Disallowed Pages and Still Indexed

    Intermediate & Advanced SEO
    3 3 2.9k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DRSearchEngOpt
      DRSearchEngOpt last edited by

      Alright, I am pretty sure I know the answer is "Nothing more I can do here." but I just wanted to double check.  It relates to the robots.txt file and that pesky "A description for this result is not available because of this site's robots.txt". Typically people want the URL indexed and the normal Meta Description to be displayed but I don't want the link there at all. I purposefully am trying to robots that stuff outta there.
      My question is, has anybody tried to get a page taken out of the Index and had this happen; URL still there but pesky robots.txt message for meta description?  Were you able to get the URL to no longer show up or did you just live with this?

      Thanks folks, you are always great!

      1 Reply Last reply Reply Quote 0
      • LoganRay
        LoganRay last edited by

        Hi Chris,

        This is a common misconception about the robots.txt file. It's not intended to remove URLs from the index, only to prevent crawling of URLs/folders/parameters/etc.

        Here's a quote straight from the Search Console Help site: "You should not use robots.txt as a means to hide your web pages from Google Search results."

        You can temporarily remove that URL using Search Console. To permanently remove it, you'll need to add a meta robots noindex tag, the only sure-fire way to prevent indexation. Here's what that tag looks like:

        Igor.Go 1 Reply Last reply Reply Quote 3
        • Igor.Go
          Igor.Go @LoganRay last edited by

          And don't forget to remove disallow in robots.txt first, if you want to remove it from index. Because if you add meta nofollow while the page is disallowed it won't go anywhere, crawler will not check it and it will stay indexed.

          Allow > Add meta noindex > wait for it to be deindexed > Disallow

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Is robots met tag a more reliable than robots.txt at preventing indexing by Google?
            Bobbi_Tschumper
            Bobbi_Tschumper
            1
            7
            3.0k

          • Will disallowing URL's in the robots.txt file stop those URL's being indexed by Google
            Martijn_Scheijbeler
            Martijn_Scheijbeler
            0
            11
            1.6k

          • If Robots.txt have blocked an Image (Image URL) but the other page which can be indexed has this image, how is the image treated?
            alphonseha
            alphonseha
            1
            4
            1.4k

          • Our client's web property recently switched over to secure pages (https) however there non secure pages (http) are still being indexed in Google. Should we request in GWMT to have the non secure pages deindexed?
            N1ghteyes
            N1ghteyes
            0
            3
            128

          • "No index" page still shows in search results and paginated pages shows page 2 in results
            khi5
            khi5
            0
            3
            114

          • How to Disallow Tag Pages With Robot.txt
            monster99
            monster99
            0
            6
            4.0k

          • Why are new pages not being indexed, and old pages (now in robots.txt) remain in the index?
            KeriMorgret
            KeriMorgret
            0
            3
            378

          • Category Pages - Canonical, Robots.txt, Changing Page Attributes
            Function5
            Function5
            0
            9
            1.1k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy