The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Robots.txt Syntax for Dynamic URLs

    Robots.txt Syntax for Dynamic URLs

    Technical SEO Issues
    5 3 369
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • btreloar
      btreloar last edited by

      I want to Disallow certain dynamic pages in robots.txt and am unsure of the proper syntax. The pages I want to disallow all include the string ?Page=

      Which is the proper syntax?
      Disallow: ?Page=
      Disallow: ?Page=*
      Disallow: ?Page=
      Or something else?

      1 Reply Last reply Reply Quote 0
      • Guest
        Guest last edited by

        This post is deleted!
        btreloar 1 Reply Last reply Reply Quote 1
        • btreloar
          btreloar @Guest last edited by

          Hi, James. It's not pagination I'm trying to disallow. The site structure has URLs that include things like "Page=give&...", that opens up a blank form ... but it comes from scores of web pages we want to spider. Since the "give" page is an empty form, we're getting tons of duplicate content errors as a result.

          1 Reply Last reply Reply Quote 0
          • Alick300
            Alick300 last edited by

            Hi Bill,

            Disallow: /?Page= will work

            Thanks

            btreloar 1 Reply Last reply Reply Quote 1
            • btreloar
              btreloar @Alick300 last edited by

              Thanks, Alick300 — unfortunately, the slash doesn't appear like that in the URLs on this site: they look like this
              www.domain.com/page.html?Page= .........

              In running through an online robots.txt tester, all three versions in my original question seem to work. Until proven otherwise, I'm using the first one because it's the simplest.

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              • First post
                Last post
              • Will a robots.txt disallow apply to a 301ed URL?
                Martijn_Scheijbeler
                Martijn_Scheijbeler
                0
                3
                158

              • Are robots.txt wildcards still valid? If so, what is the proper syntax for setting this up?
                DarinPirkey
                DarinPirkey
                0
                3
                7.0k

              • Blocked URL's by robots.txt
                meralucian37
                meralucian37
                0
                7
                398

              • Question about construction of our sitemap URL in robots.txt file
                danatanseo
                danatanseo
                0
                16
                1.6k

              • Robots.txt - What is the correct syntax?
                teleman
                teleman
                0
                6
                297

              • Can I Disallow Faceted Nav URLs - Robots.txt
                AlanMosley
                AlanMosley
                0
                5
                914

              • Robots.txt Syntax
                dohertyjf
                dohertyjf
                0
                4
                766

              • Does RogerBot read URL wildcards in robots.txt
                jennita
                jennita
                0
                2
                928

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy