The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. On-Page / Site Optimization
    4. Question about robots.txt

    Question about robots.txt

    On-Page / Site Optimization
    3 3 125
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • mozismoz
      mozismoz last edited by

      Solved!

      1 Reply Last reply Reply Quote 0
      • WilliamKammer
        WilliamKammer last edited by

        Consider deleting all of this:

        Disallow: /&limit
        Disallow: /
        ?limit
        Disallow: /&sort
        Disallow: /
        ?sort
        Disallow: /?route=checkout/
        Disallow: /
        ?route=account/
        Disallow: /?route=product/search
        Disallow: /
        ?route=affiliate/
        Disallow: /?marca
        Disallow: /
        &manufacturer
        Disallow: /?manufacturer
        Disallow: /
        ?filter
        Disallow: /&filter
        Disallow: /
        ?order
        Disallow: /&order
        Disallow: /
        ?price
        Disallow: /&price
        Disallow: /
        ?filter_tag
        Disallow: /&filter_tag
        Disallow: /
        ?mode
        Disallow: /&mode
        Disallow: /
        ?cat
        Disallow: /&cat
        Disallow: /
        ?product_id
        Disallow: /&product_id
        Disallow: /
        ?route=affiliate/
        Disallow: /*?keyword

        Those rules are telling Google not to crawl domain.com/EVERYTHING(then the URL parameter). This could be where the issue stems from. If you're worried about URLs with these things ranking, consider implementing canonical tags instead to point to the proper pages

        1 Reply Last reply Reply Quote 1
        • spencerhjustice
          spencerhjustice last edited by

          Just a friendly reminder. Please don't delete your question after it's been answered. It's very likely that someone in the future could have the same question and they would have been able to find the answer if you hadn't deleted the question.

          1 Reply Last reply Reply Quote 2
          • 1 / 1
          • First post
            Last post
          • Correct robots.txt for WordPress
            jasongmcmahon
            jasongmcmahon
            1
            5
            180

          • Robots.txt Question for E-Commerce Sites
            Joe.Robison
            Joe.Robison
            0
            2
            450

          • Two Robots.txt files
            TrulyTravel
            TrulyTravel
            0
            5
            514

          • Robot.txt file issue on wordpress site.
            AlanMosley
            AlanMosley
            0
            8
            229

          • Can Robots.txt on Root Domain override a Robots.txt on a Sub Domain?
            RyanPurkey
            RyanPurkey
            0
            3
            522

          • How do i block an entire category/directory with robots.txt?
            bricerhodes
            bricerhodes
            0
            3
            2.7k

          • Wordpress categories tags and robots.txt
            sfmatthews
            sfmatthews
            0
            4
            714

          • Robots.txt file
            Greenman
            Greenman
            0
            3
            652

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy