The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Search Engine Blocked by Robot Txt warnings for Filter Search result pages--Why?

    Search Engine Blocked by Robot Txt warnings for Filter Search result pages--Why?

    Technical SEO Issues
    5 3 915
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • languedoc
      languedoc last edited by

      Hi,

      We're getting 'Yellow' Search Engine Blocked by Robot Txt warnings for URLS that are in effect product search filter result pages (see link below) on our Magento ecommerce shop. Our Robot txt file to my mind is correctly set up i.e. we would not want Google to index these pages. So why does SeoMoz flag this type of page as a warning? Is there any implication for our ranking? Is there anything we need to do about this? Thanks.

      Here is an example url that SEOMOZ thinks that the search engines can't see.

      http://www.site.com/audio-books/audio-books-in-english?audiobook_genre=132

      Below are the current entries for the robot.txt file.

      User-agent: Googlebot
      Disallow: /index.php/
      Disallow: /?
      Disallow: /
      .js$
      Disallow: /.css$
      Disallow: /checkout/
      Disallow: /tag/
      Disallow: /catalogsearch/
      Disallow: /review/
      Disallow: /app/
      Disallow: /downloader/
      Disallow: /js/
      Disallow: /lib/
      Disallow: /media/
      Disallow: /
      .php$
      Disallow: /pkginfo/
      Disallow: /report/
      Disallow: /skin/
      Disallow: /utm
      Disallow: /var/
      Disallow: /catalog/
      Disallow: /customer/
      Sitemap:

      1 Reply Last reply Reply Quote 0
      • rickdronkers
        rickdronkers last edited by

        So what your saying is:

        1. SEOmoz says these pages can't get indexed by search engines because of our robot.txt

        2. We don't want these pages indexed and blocked them using robots.txt

        My initial reaction is: no problem, SEOmoz is just showing you as a 'confirmation warning' that these pages are not indexed, but since you did that on purpose, it's okay.

        Hope this helps!

        languedoc 1 Reply Last reply Reply Quote 1
        • KeriMorgret
          KeriMorgret last edited by

          Like Rick said, it's just a "hey, make sure that you really wanted to do this" type warning, since you can easily write a robots.txt that blocks things you didn't really think would be blocked. Or someone else can modify the robots.txt without telling you, and this can be a warning that you need to go find someone and get that fixed.

          languedoc 1 Reply Last reply Reply Quote 0
          • languedoc
            languedoc @rickdronkers last edited by

            Thanks Rick for your advice

            1 Reply Last reply Reply Quote 0
            • languedoc
              languedoc @KeriMorgret last edited by

              Thanks Keri for your advice

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              • First post
                Last post
              • 3,511 Pages Indexed and 3,331 Pages Blocked by Robots
                PeaSoupDigital
                PeaSoupDigital
                0
                4
                114

              • HTTP Status showing up in opensiteexplorer top pages as blocked by robot.txt file
                FedeEinhorn
                FedeEinhorn
                0
                2
                106

              • "Extremely high number of URLs" warning for robots.txt blocked pages
                KristinaKledzik
                KristinaKledzik
                0
                8
                380

              • Block or remove pages using a robots.txt
                OlegKorneitchouk
                OlegKorneitchouk
                0
                2
                422

              • Search/Search Results Page & Duplicate Content
                AJPro
                AJPro
                0
                2
                272

              • Warnings for blocked by blocked by meta-robots/meta robots Nofollow...how to resolve?
                Cyrus-Shepard
                Cyrus-Shepard
                0
                3
                415

              • Do you get credit for an external link that points to a page that's being blocked by robots.txt
                ShaMenz
                ShaMenz
                0
                2
                390

              • Search engines have been blocked by robots.txt., how do I find and fix it?
                KeriMorgret
                KeriMorgret
                0
                7
                524

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy