The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Clarification regarding robots.txt protocol

    Clarification regarding robots.txt protocol

    Technical SEO Issues
    4 4 118
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • nlogix
      nlogix last edited by

      Hi, 
      I have a website , and having 1000 above url and all the url already got indexed in Google . Now am going to stop all the available services in my website and removed all the landing pages from website. Now only home page available . So i need to remove all the indexed urls from Google . I have already used robots txt protocol for removing url. i guess it is not a good method for adding bulk amount of urls (nearly 1000) in robots.txt . So just wanted to know is there any other method for removing indexed urls. 
      Please advice.

      1 Reply Last reply Reply Quote 0
      • MoosaHemani
        MoosaHemani last edited by

        If the target is to get the URLs out of the search engine index than there are the few solutions can work for you:

        1. The one your mentioned: I think it’s bad to add 1000+ URLs in robots.txt file its make sense for your business.
        2. Adding meta no-index tag to the pages (if pages physically exist).

        Also in order to quickly remove them from the index you can update robots.txt file and then go to GWC and use remove URL feature.

        Just a thought!

        1 Reply Last reply Reply Quote 1
        • GlobeRunner
          GlobeRunner last edited by

          There are a few ways to do this.

          First, I would use the Google Removal Tool to remove those URLs. More information here: https://support.google.com/webmasters/answer/1663419?hl=en

          Then, using the robots.txt file is good, you need to make sure that you're listing the correct URLs or URL path there.

          I would make sure that you are using a "410 Gone" in the server header, and not a 404 error. The 410  Gone will get those URLs removed faster.

          1 Reply Last reply Reply Quote 1
          • OlegKorneitchouk
            OlegKorneitchouk last edited by

            If the pages are already indexed and you want them to be completely removed, you need to allow the crawlers in robots.txt and noindex the individual pages.

            So if you just block the site with robots.txt (and I recommend blocking via folders or variables, not individual pages) while the pages are indexed, they will continue to appear in search results but have a meta description of (this page is being blocked by robots.txt). However, it will continue to rank and appear because of the cached data.

            If you add the noindex tags to your pages instead, the next time crawlers visit the pages they will see the new tag and remove the page from the search index (meaning it won't show up at all). However, make sure your robots.txt isn't blocking the crawlers from seeing this updated code.

            1 Reply Last reply Reply Quote 2
            • 1 / 1
            • First post
              Last post
            • Robots.txt
              MarieHaynes
              MarieHaynes
              0
              8
              115

            • Robots.txt
              MichaelC-15022
              MichaelC-15022
              0
              7
              1.0k

            • Meta Robots Noindex and Robots.txt File
              Devanur-Rafi
              Devanur-Rafi
              0
              2
              125

            • Robots.txt
              irvingw
              irvingw
              0
              4
              116

            • Robots.txt file
              Asher
              Asher
              0
              3
              261

            • Do I need robots.txt and meta robots?
              Cyrus-Shepard
              Cyrus-Shepard
              0
              7
              1.1k

            • Robots.txt
              Ontarioseo
              Ontarioseo
              0
              5
              737

            • Robots.txt and robots meta
              TheEspresseo
              TheEspresseo
              0
              5
              1.1k

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy