The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Need help with Robots.txt

    Need help with Robots.txt

    Intermediate & Advanced SEO
    3 3 127
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Nahid
      Nahid last edited by

      An eCommerce site built with Modx CMS. I found lots of auto generated duplicate page issue on that site. Now I need to disallow some pages from that category. Here is the actual product page url looks like
      product_listing.php?cat=6857

      And here is the auto generated url structure
      product_listing.php?cat=6857&cPath=dropship&size=19

      Can any one suggest how to disallow this specific category through robots.txt.  I am not so familiar with Modx and this kind of link structure.

      Your help will be appreciated.

      Thanks

      1 Reply Last reply Reply Quote 1
      • GlobeRunner
        GlobeRunner last edited by

        Nahid, before you use the robots.txt file's disallow for those URLs, you may want to reconsider. You may want to use the canonical tag instead. In the case where you have different sizes, colors, etc. we typically recommend using the Canonical Tag and not the disallow in robots.txt.

        Anyhow, if you'd like to use the disallow you can use one of these:

        Disallow: /?

        or

        Disallow: /?cat=

        1 Reply Last reply Reply Quote 2
        • MattAntonino
          MattAntonino last edited by

          I would actually add a canonical tag and then handle these using the Parameters section of Search Console. That's why it's there, for exactly this type of site with exactly this issue.

          https://support.google.com/webmasters/answer/6080550?hl=en

          1 Reply Last reply Reply Quote 2
          • 1 / 1
          • First post
            Last post
          • Robots.txt Help
            GlobeRunner
            GlobeRunner
            0
            5
            162

          • Robots.txt Allowed
            GlobeRunner
            GlobeRunner
            0
            4
            118

          • I need help!
            HillyCE
            HillyCE
            0
            5
            119

          • Avoiding Duplicate Content with Used Car Listings Database: Robots.txt vs Noindex vs Hash URLs (Help!)
            Everett
            Everett
            0
            10
            2.0k

          • Robots.txt, does it need preceding directory structure?
            Milian
            Milian
            0
            3
            156

          • Robot.txt help
            evolvingSEO
            evolvingSEO
            0
            23
            203

          • Robot.txt error
            Rubix
            Rubix
            0
            13
            203

          • Help needed for a domain
            smarties954
            smarties954
            0
            2
            103

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy