The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Question about Syntax in Robots.txt

    Question about Syntax in Robots.txt

    Intermediate & Advanced SEO
    7 4 154
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DRSearchEngOpt
      DRSearchEngOpt last edited by

      So if I want to block any URL from being indexed that contains a particular parameter what is the best way to put this in the robots.txt file?

      Currently I have-
      Disallow: /attachment_id

      Where "attachment_id" is the parameter.  Problem is I still see these URL's indexed and this has been in the robots now for over a month. I am wondering if I should just do

      Disallow: attachment_id or Disallow: attachment_id= but figured I would ask you guys first.

      Thanks!

      1 Reply Last reply Reply Quote 0
      • ATP
        ATP last edited by

        Im not a robots.txt expert by a long shot, but I found this, which is a little dated, which explained it to me in terms i could understand.

        https://sanzon.wordpress.com/2008/04/29/advanced-usage-of-robotstxt-w-querystrings/

        there is also a feature in Google Webmaster tools  called URL parameters that lets you block URLs with set parameters for all sorts of reason to avoid duplicate content etc. I havn't used it myself but may be work looking into

        https://support.google.com/webmasters/answer/6080550?hl=en

        1 Reply Last reply Reply Quote 1
        • PatrickDelehanty
          PatrickDelehanty last edited by

          Hi there

          Take a look at Google's resource on robots.txt, as well as Moz's. You can get all the information you need there. You can also let Google know about what URLs to exclude from it's crawls via Search Console.

          Hope this helps! Good luck!

          DRSearchEngOpt 1 Reply Last reply Reply Quote 1
          • Andy.Drinkwater
            Andy.Drinkwater last edited by

            Problem is I still see these URL's indexed and this has been in the robots now for over a month. I am wondering if I should just do

            It can take Google some time to remove pages from the index.

            The best way to test if this has worked is hop into Webmaster Tools and use the Test Robots.txt function. If it has blocked the required pages, then you know it's just a case of waiting - you can also remove pages from within Webmaster Tools as well, although this isn't immediate.

            -Andy

            DRSearchEngOpt 1 Reply Last reply Reply Quote 1
            • DRSearchEngOpt
              DRSearchEngOpt @PatrickDelehanty last edited by

              I have previously looked into both of those documents and the issue remains that they don't exactly address how best to block parameters.  I could do this through GWT but just am curious about the correct and preferred syntax for the robots.txt as well.  I guess I could just look at sites like Amazon or other big sites to see what the common practices are.  Thanks though!

              1 Reply Last reply Reply Quote 0
              • DRSearchEngOpt
                DRSearchEngOpt @Andy.Drinkwater last edited by

                I don't know how but I completely forgot I could just pop those URL's in GWT and see if they were blocked or not and sure enough, Google says they are.  I guess this is just a matter of waiting.... Thanks much!

                Andy.Drinkwater 1 Reply Last reply Reply Quote 0
                • Andy.Drinkwater
                  Andy.Drinkwater @DRSearchEngOpt last edited by

                  That's excellent Chris.

                  Use the Remove Page function as well - it might help speed things up for you.

                  -Andy

                  1 Reply Last reply Reply Quote 0
                  • 1 / 1
                  • First post
                    Last post
                  • Robots.txt & Disallow: /*? Question!
                    BabaBha0173
                    BabaBha0173
                    0
                    8
                    321

                  • Google robots.txt test - not picking up syntax errors?
                    McTaggart
                    McTaggart
                    0
                    3
                    108

                  • Meta robots or robot.txt file?
                    Andy.Drinkwater
                    Andy.Drinkwater
                    0
                    5
                    152

                  • Robots.txt
                    Travis_Bailey
                    Travis_Bailey
                    0
                    4
                    107

                  • Robots.txt help
                    KeriMorgret
                    KeriMorgret
                    0
                    4
                    85

                  • Robots.txt
                    TomRayner
                    TomRayner
                    0
                    5
                    137

                  • Robot.txt help
                    evolvingSEO
                    evolvingSEO
                    0
                    23
                    203

                  • Robots.txt: Syntax URL to disallow
                    Anti-Alex
                    Anti-Alex
                    0
                    8
                    479

                  Get started with Moz Pro!

                  Unlock the power of advanced SEO tools and data-driven insights.

                  Start my free trial
                  Products
                  • Moz Pro
                  • Moz Local
                  • Moz API
                  • Moz Data
                  • STAT
                  • Product Updates
                  Moz Solutions
                  • SMB Solutions
                  • Agency Solutions
                  • Enterprise Solutions
                  • Digital Marketers
                  Free SEO Tools
                  • Domain Authority Checker
                  • Link Explorer
                  • Keyword Explorer
                  • Competitive Research
                  • Brand Authority Checker
                  • Local Citation Checker
                  • MozBar Extension
                  • MozCast
                  Resources
                  • Blog
                  • SEO Learning Center
                  • Help Hub
                  • Beginner's Guide to SEO
                  • How-to Guides
                  • Moz Academy
                  • API Docs
                  About Moz
                  • About
                  • Team
                  • Careers
                  • Contact
                  Why Moz
                  • Case Studies
                  • Testimonials
                  Get Involved
                  • Become an Affiliate
                  • MozCon
                  • Webinars
                  • Practical Marketer Series
                  • MozPod
                  Connect with us

                  Contact the Help team

                  Join our newsletter
                  Moz logo
                  © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                  • Accessibility
                  • Terms of Use
                  • Privacy