The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Robots.txt Syntax

    Robots.txt Syntax

    Intermediate & Advanced SEO
    2 2 118
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DRSearchEngOpt
      DRSearchEngOpt last edited by

      I have been having a hard time finding any decent information regarding the robots.txt syntax that has been written in the last few years and I just want to verify some things as a review for myself.  I have many occasions where I need to block particular directories in the URL, parameters and parameter values.  I just wanted to make sure that I am doing this in the most efficient ways possible and thought you guys could help.

      So let's say I want to block a particular directory called "this" and this would be an example URL:

      www.domain.com/folder1/folder2/this/file.html  
      or
      www.domain.com/folder1/this/folder2/file.html

      In order for me to block any URL that contains this folder anywhere in the URL I would use:

      User-agent: *
      Disallow: /this/

      Now lets say I have a parameter "that" I want to block and sometimes it is the first parameter and sometimes it isn't when it shows up in the URL.  Would it look like this?

      User-agent: *
      Disallow: ?that=
      Disallow: &that=

      What about if there is only one value I want to block for "that" and the value is "NotThisGuy":

      User-agent: *
      Disallow: ?that=NotThisGuy
      Disallow: &that=NotThisGuy

      My big questions here are what are the most efficient ways to block a particular parameter and block a particular parameter value.  Is there a more efficient way to deal with ? and & for when the parameter and value are either first or later?  Secondly is there a list somewhere that will tell me all of the syntax and meaning that can be used for a robots.txt file?

      Thanks!

      1 Reply Last reply Reply Quote 0
      • MichaelC-15022
        MichaelC-15022 last edited by

        My advice is to go easy with robots.txt--it's a bit like dynamite, powerful, but can take your leg (or entire website) off.

        I like this checker:

        http://tool.motoricerca.info/robots-checker.phtml

        If you look ok after running that checker, then use the built-in Google one.

        Note that robots.txt syntax DOES NOT have wildcards.  Apparently this doesn't stop a ton of people from using wildcards in them (to no effect, and clearly they didn't bother to test!).

        Another reason to avoid disallow in robots.txt is that if you disallow the engines from looking at a page's contents, then you're ALSO stopping the link juice that might have flowed to other pages it links to.

        So let's say you have 100 pages on your site that you're currently blocking with disallow in robots.txt.  If instead, you put a meta robots "noindex,follow" in each of those pages, then every page linked to from those 100 pages (i.e. everything in your main menu) would get an extra 100 internal links worth of link juice.

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        • Is robots met tag a more reliable than robots.txt at preventing indexing by Google?
          Bobbi_Tschumper
          Bobbi_Tschumper
          1
          7
          3.0k

        • Robots.txt Allowed
          GlobeRunner
          GlobeRunner
          0
          4
          118

        • Meta robots or robot.txt file?
          Andy.Drinkwater
          Andy.Drinkwater
          0
          5
          152

        • Question about Syntax in Robots.txt
          Andy.Drinkwater
          Andy.Drinkwater
          0
          7
          154

        • Robots.txt
          Travis_Bailey
          Travis_Bailey
          0
          4
          107

        • Robots.txt help
          KeriMorgret
          KeriMorgret
          0
          4
          85

        • Robots.txt assistance
          theLotter
          theLotter
          0
          9
          280

        • Robots.txt
          TomRayner
          TomRayner
          0
          5
          137

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy