The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. I want to block search bots in crawling all my website's pages expect for homepage. Is this rule correct?

    I want to block search bots in crawling all my website's pages expect for homepage. Is this rule correct?

    Intermediate & Advanced SEO
    7 5 135
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • esiow2013
      esiow2013 last edited by

      User-agent: *

      Disallow: /*

      1 Reply Last reply Reply Quote 0
      • Travis_Bailey
        Travis_Bailey last edited by

        I don't think that will work. Robots.txt doesn't handle regular expressions. You will have to explicitly list all of the folders, and files to be super sure, that nothing is indexed unless you want it to be found.

        This is kind of an odd question. I haven't thought about something like this in a while. I usually want everything but a couple folders indexed. : ) I found something that may be a little more help. Try reading this.

        If you're working with extensions, you can use **Disallow:/*.html$ **or php or what have you. That may get you closer to a solution.

        Definitely test this with a crawler that obeys robots.txt.

        1 Reply Last reply Reply Quote 0
        • MattAntonino
          MattAntonino last edited by

          Actually, there are two regex that Robots can handle - asterisk and $.

          You should test this one.  I think it will work (about 95% sure - tested in WMT quickly):

          User-agent: *
          Disallow: /
          Allow: /$

          esiow2013 1 Reply Last reply Reply Quote 0
          • Kingof5
            Kingof5 last edited by

            Use this:

            User-agent: Googlebot
            Noindex: /

            User-agent: Googlebot
            Disallow: /

            User-agent: *
            Disallow: /

            This is what I use to block our dev sites from being indexed and we've had no issues.

            esiow2013 1 Reply Last reply Reply Quote 0
            • esiow2013
              esiow2013 @Kingof5 last edited by

              Thanks David! Will try this one.

              1 Reply Last reply Reply Quote 0
              • esiow2013
                esiow2013 @MattAntonino last edited by

                Thanks Matt! I will surely test this one.

                1 Reply Last reply Reply Quote 0
                • GPainter
                  GPainter last edited by

                  some great answers you can also find a list of all the robots here & here  

                  Depending on your site you can also for example hide the rest of your site behind a login screen or a form which bots won't fill in.

                  1 Reply Last reply Reply Quote 0
                  • 1 / 1
                  • First post
                    Last post
                  • A client rebranded a few years ago and doesn't want to be associated with it's old brand name. He wishes not to appear when the old brand is searched in Google, is there something we can do?
                    0
                    1
                    28

                  • How will changing my website's page content affect SEO?
                    WebMarkets
                    WebMarkets
                    1
                    5
                    3.4k

                  • How to remove my site's pages in search results?
                    CleverPhD
                    CleverPhD
                    0
                    7
                    108

                  • What to do when you buy a Website without it's content which has a few thousand pages indexed?
                    BeanstalkIM
                    BeanstalkIM
                    0
                    5
                    132

                  • No matter what I do, my website isn't showing up in search results. What's happening?
                    irvingw
                    irvingw
                    0
                    5
                    311

                  • Soft 404's from pages blocked by robots.txt -- cause for concern?
                    CleverPhD
                    CleverPhD
                    1
                    6
                    2.0k

                  • Does Google crawl the pages which are generated via the site's search box queries?
                    YannickVeys
                    YannickVeys
                    0
                    5
                    606

                  • Don't want to lose page rank, what's the best way to restructure a url other than a 301 redirect?
                    marig
                    marig
                    0
                    6
                    1.0k

                  Get started with Moz Pro!

                  Unlock the power of advanced SEO tools and data-driven insights.

                  Start my free trial
                  Products
                  • Moz Pro
                  • Moz Local
                  • Moz API
                  • Moz Data
                  • STAT
                  • Product Updates
                  Moz Solutions
                  • SMB Solutions
                  • Agency Solutions
                  • Enterprise Solutions
                  • Digital Marketers
                  Free SEO Tools
                  • Domain Authority Checker
                  • Link Explorer
                  • Keyword Explorer
                  • Competitive Research
                  • Brand Authority Checker
                  • Local Citation Checker
                  • MozBar Extension
                  • MozCast
                  Resources
                  • Blog
                  • SEO Learning Center
                  • Help Hub
                  • Beginner's Guide to SEO
                  • How-to Guides
                  • Moz Academy
                  • API Docs
                  About Moz
                  • About
                  • Team
                  • Careers
                  • Contact
                  Why Moz
                  • Case Studies
                  • Testimonials
                  Get Involved
                  • Become an Affiliate
                  • MozCon
                  • Webinars
                  • Practical Marketer Series
                  • MozPod
                  Connect with us

                  Contact the Help team

                  Join our newsletter
                  Moz logo
                  © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                  • Accessibility
                  • Terms of Use
                  • Privacy