The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Robots.txt file - How to block thosands of pages when you don't have a folder path

    Robots.txt file - How to block thosands of pages when you don't have a folder path

    Intermediate & Advanced SEO
    4 2 264
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Unity
      Unity last edited by

      Hello.
      Just wondering if anyone has come across this and can tell me if it worked or not.

      Goal:
      To block review pages

      Challenge:
      The URLs aren't constructed using folders, they look like this:
      www.website.com/default.aspx?z=review&PG1234
      www.website.com/default.aspx?z=review&PG1235
      www.website.com/default.aspx?z=review&PG1236

      So the first part of the URL is the same (i.e. /default.aspx?z=review) and the unique part comes immediately after - so not as a folder. Looking at Google recommendations they show examples for ways to block 'folder directories' and 'individual pages' only.

      Question:
      If I add the following to the Robots.txt file will it block all review pages?

      User-agent: *
      Disallow: /default.aspx?z=review

      Much thanks,
      Davinia

      1 Reply Last reply Reply Quote 0
      • Klarke
        Klarke last edited by

        http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449

        Head down to the pattern matching section.

        I think

        User-agent: *
        Disallow: /default.aspx?z=review*

        should do the trick though.

        Unity 1 Reply Last reply Reply Quote 1
        • Unity
          Unity @Klarke last edited by

          An * added to the end! Great thank you!

          1 Reply Last reply Reply Quote 0
          • Klarke
            Klarke last edited by

            Also remember that blocking in robots.txt doesn't prevent Google from indexing those URLs.  If the URLs are already indexed or if they are linked to, either internally or externally they may still in appear in the index with limited snippet information.  If so, you'll need to add a noindex meta tag to those pages.

            1 Reply Last reply Reply Quote 1
            • 1 / 1
            • First post
              Last post
            • Magento 1.9 SEO. I have product pages with identical On Page SEO score in the 90's. Some pull up Google page 1 some won't pull up at all. I am searching for the exact title on that page.
              CTOPDS
              CTOPDS
              0
              3
              63

            • Search Results Pages Blocked in Robots.txt?
              BeckyKey
              BeckyKey
              0
              3
              117

            • Pagination and View All Pages Question. We currently don't have a canonical tag pointing to View all as I don't believe it's a good user experience so how best we deal with this.
              PeteC12
              PeteC12
              0
              3
              117

            • Using folder blocked by robots.txt before uploaded to indexed folder - is that OK?
              khi5
              khi5
              0
              4
              100

            • Pages getting into Google Index, blocked by Robots.txt??
              Devanur-Rafi
              Devanur-Rafi
              0
              10
              673

            • I've seen and heard alot about city-specific landing pages for businesses with multiple locations, but what about city-specific landing pages for cities nearby that you aren't actually located in? Is it ok to create landing pages for nearby cities?
              AlanMosley
              AlanMosley
              1
              3
              1.9k

            • Issue with Robots.txt file blocking meta description
              ThompsonPaul
              ThompsonPaul
              0
              9
              3.8k

            • Does using robots.txt to block pages decrease search traffic?
              KeriMorgret
              KeriMorgret
              0
              4
              520

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy