The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. How to remove my site's pages in search results?

    How to remove my site's pages in search results?

    Intermediate & Advanced SEO
    7 5 108
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • esiow2013
      esiow2013 last edited by

      I have tested hundreds of pages to see if Google will properly crawl, index and cached them. Now, I want these pages to be removed in Google search except for homepage. What should be the rule in robots.txt?

      I use this rule, but I am not sure if Google will remove the hundreds of pages (for my testing).

      User-agent: *
      Disallow: /
      Allow: /$

      1 Reply Last reply Reply Quote 0
      • MattAntonino
        MattAntonino last edited by

        This is basically a duplicate of your other thread where I gave you that code.  Yes, it should block the other pages.  Put that in, fetch in WMT and you should be right.

        Also, you can test it before you implement in WMT as well. I tried it on my end and it works.

        esiow2013 1 Reply Last reply Reply Quote 1
        • maxweb
          maxweb last edited by

          I would do

          User-agent: *
          Disallow: /?
          Allow: /

          But test it first in WMT first to be safe. However, you must be sure that this is the route you want to go down. Robots.txt will prevent all of those pages from being indexed, which means that none of their content will count. Any links to these pages may also be devalued. The result is a potential drop in SERPs.

          What is the reason why you don't want them appearing? That way we may find an alternative solution.

          esiow2013 1 Reply Last reply Reply Quote 0
          • esiow2013
            esiow2013 @maxweb last edited by

            These are just test pages and I need them to be private and not visible in Google after I test. I understand that there will be a drop in SERP rankings.

            1 Reply Last reply Reply Quote 0
            • datawork
              datawork last edited by

              why don't you try adding a meta robots tag on those pages with "NOINDEX".

              i would also do remove url with WMT

              1 Reply Last reply Reply Quote 0
              • esiow2013
                esiow2013 @MattAntonino last edited by

                Hi Matt! I've already tried your suggestion. I'll let you know what's the result. Thanks a lot man!

                1 Reply Last reply Reply Quote 0
                • CleverPhD
                  CleverPhD last edited by

                  Why not just 404/410 those pages?

                  1 Reply Last reply Reply Quote 0
                  • 1 / 1
                  • First post
                    Last post
                  • Magento 1.9 SEO. I have product pages with identical On Page SEO score in the 90's. Some pull up Google page 1 some won't pull up at all. I am searching for the exact title on that page.
                    CTOPDS
                    CTOPDS
                    0
                    3
                    63

                  • When Mobile and Desktop sites have the same page URLs, how should I handle the 'View Desktop Site' link on a mobile site to ensure a smooth crawl?
                    DirkC
                    DirkC
                    0
                    3
                    1.4k

                  • Does Google still don't index Hashtag Links ? No chance to get a Search Result that leads directly to a section of a page? or to one of numeras Hashtag Pages in a single HTML page?
                    Muhammad_Jabali
                    Muhammad_Jabali
                    0
                    3
                    748

                  • "No index" page still shows in search results and paginated pages shows page 2 in results
                    khi5
                    khi5
                    0
                    3
                    114

                  • What's the best way to check Google search results for all pages NOT linking to a domain?
                    Cyrus-Shepard
                    Cyrus-Shepard
                    0
                    7
                    163

                  • No matter what I do, my website isn't showing up in search results. What's happening?
                    irvingw
                    irvingw
                    0
                    5
                    311

                  • Will pages irrelevant to a site's core content dilute SEO value of core pages?
                    ArchMedia
                    ArchMedia
                    0
                    4
                    486

                  • Is 404'ing a page enough to remove it from Google's index?
                    RyanKent
                    RyanKent
                    0
                    4
                    13.7k

                  Get started with Moz Pro!

                  Unlock the power of advanced SEO tools and data-driven insights.

                  Start my free trial
                  Products
                  • Moz Pro
                  • Moz Local
                  • Moz API
                  • Moz Data
                  • STAT
                  • Product Updates
                  Moz Solutions
                  • SMB Solutions
                  • Agency Solutions
                  • Enterprise Solutions
                  • Digital Marketers
                  Free SEO Tools
                  • Domain Authority Checker
                  • Link Explorer
                  • Keyword Explorer
                  • Competitive Research
                  • Brand Authority Checker
                  • Local Citation Checker
                  • MozBar Extension
                  • MozCast
                  Resources
                  • Blog
                  • SEO Learning Center
                  • Help Hub
                  • Beginner's Guide to SEO
                  • How-to Guides
                  • Moz Academy
                  • API Docs
                  About Moz
                  • About
                  • Team
                  • Careers
                  • Contact
                  Why Moz
                  • Case Studies
                  • Testimonials
                  Get Involved
                  • Become an Affiliate
                  • MozCon
                  • Webinars
                  • Practical Marketer Series
                  • MozPod
                  Connect with us

                  Contact the Help team

                  Join our newsletter
                  Moz logo
                  © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                  • Accessibility
                  • Terms of Use
                  • Privacy