The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Blocking out specific URLs with robots.txt

    Blocking out specific URLs with robots.txt

    Intermediate & Advanced SEO
    3 3 133
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Whebb
      Whebb last edited by

      I've been trying to block out a few URLs using robots.txt, but I can't seem to get the specific one I'm trying to block. Here is an example.

      I'm trying to block

      something.com/cats

      but not block

      something.com/cats-and-dogs

      It seems if it setup my robots.txt as so..

      Disallow: /cats

      It's blocking both urls. When I crawl the site with screaming flog, that Disallow is causing both urls to be blocked.  How can I set up my robots.txt to specifically block /cats? I thought it was by doing it the way I was, but that doesn't seem to solve it.

      Any help is much appreciated, thanks in advance.

      1 Reply Last reply Reply Quote 0
      • LesleyPaone
        LesleyPaone last edited by

        You can either use /cats/ or /cats/* that should just block the cats folder and not the other folder. Note the first use is the preferred one.

        1 Reply Last reply Reply Quote 1
        • Modi
          Modi last edited by

          Do not play with Robots as it may block out series of pages and folders out of index

          Correct command as stated by Lesley is /cats/ . Refer official documentation

          https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Block session id URLs with robots.txt
            Mat_C
            Mat_C
            1
            4
            130

          • Will disallowing URL's in the robots.txt file stop those URL's being indexed by Google
            Martijn_Scheijbeler
            Martijn_Scheijbeler
            0
            11
            1.6k

          • If Robots.txt have blocked an Image (Image URL) but the other page which can be indexed has this image, how is the image treated?
            alphonseha
            alphonseha
            1
            4
            1.4k

          • Robots.txt Blocked Most Site URLs Because of Canonical
            0
            1
            117

          • Can URLs blocked with robots.txt hurt your site?
            workzentre
            workzentre
            0
            4
            302

          • Why are these results being showed as blocked by robots.txt?
            eyepaq
            eyepaq
            0
            9
            203

          • Block an entire subdomain with robots.txt?
            kylesuss
            kylesuss
            1
            16
            102.1k

          • Should we block urls like this - domainname/shop/leather-chairs.html?brand=244&cat=16&dir=ascℴ=price&price=1 within the robots.txt?
            sferrino
            sferrino
            0
            2
            864

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy