The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Search Engine Blocked by robots.txt for Dynamic URLs

    Search Engine Blocked by robots.txt for Dynamic URLs

    Intermediate & Advanced SEO
    2 2 689
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • CommercePundit
      CommercePundit last edited by

      Today, I was checking crawl diagnostics for my website. I found warning for search engine blocked by robots.txt

      I have added following syntax to robots.txt file for all dynamic URLs.

      Disallow: /*?osCsid

      Disallow: /*?q=

      Disallow: /*?dir=

      Disallow: /*?p=

      Disallow: /*?limit=

      Disallow: /*review-form

      Dynamic URLs are as follow.

      http://www.vistastores.com/bar-stools?dir=desc&order=position

      http://www.vistastores.com/bathroom-lighting?p=2

      and many more...

      So, Why should it shows me warning for this? Does it really matter or any other solution for these kind of dynamic URLs.

      1 Reply Last reply Reply Quote 0
      • KeriMorgret
        KeriMorgret last edited by

        It's a warning, not an error. It means "hey, make sure that you really want this stuff blocked" not "you made a mistake". And yes, you really do want those annoying osCsid parameters blocked!

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        • Search engine blocked by robots-crawl error by moz & GWT
          RuthBurrReedy
          RuthBurrReedy
          0
          11
          445

        • Robots.txt Blocked Most Site URLs Because of Canonical
          0
          1
          117

        • Will blocking urls in robots.txt void out any backlink benefits? - I'll explain...
          AubieJon
          AubieJon
          0
          4
          1.0k

        • Should Site Search results be blocked from search engines?
          wissamdandan
          wissamdandan
          0
          2
          175

        • Block all but one URL in a directory using robots.txt?
          Cyrus-Shepard
          Cyrus-Shepard
          0
          3
          3.5k

        • Does using robots.txt to block pages decrease search traffic?
          KeriMorgret
          KeriMorgret
          0
          4
          520

        • Does It Really Matter to Restrict Dynamic URLs by Robots.txt?
          CommercePundit
          CommercePundit
          0
          4
          902

        • Block all search results (dynamic) in robots.txt?
          onwebtoday
          onwebtoday
          0
          9
          4.8k

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy