The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Other Questions
    4. How do I disallow MOZ from crawling URL's with parameters?

    How do I disallow MOZ from crawling URL's with parameters?

    Other Questions
    3 3 340
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Revolution-Motorsports
      Revolution-Motorsports last edited by

      I'm seeing hundreds of pages with issues on my crawl reports. When I research the URL's, I see they are all duplicates of the same URL's, with parameters following the question mark symbol.

      I have already added the URL Parameter rules to Google Webmaster Tools, so Google knows how to handle them.

      Does MOZ have a similar setting that I can edit?

      1 Reply Last reply Reply Quote 0
      • PatrickDelehanty
        PatrickDelehanty last edited by

        Hi there

        Don't forget your canonical tags! You should read through Google's resource on duplicate content as well.

        You can block this access with your robots.txt by using rogetbot as your user agent.

        You can read more from this Q+A thread as well.

        Hope this helps! Good luck!

        1 Reply Last reply Reply Quote 2
        • Dezzign
          Dezzign last edited by

          Yes definitely don't forget canonical tags where possible. But I think those pages are being auto generated  via searches on your website or some other means. Most good Seo plugins or extensions prevent these pages from being indexed. Second to that, find the parent page and use use a wildcard block command in your robots.txt file. It should look something like this:

          User-agent: *
          Disallow: /your-inner-page/*

          That command will block every url variation after www.yoursite.com/your-inner-page/ from being indexed or crawled but still allow your real page url of www.yoursite.com/your-inner-page/  to be crawled and indexed.

          1 Reply Last reply Reply Quote 1
          • 1 / 1
          • First post
            Last post
          • Too Many Moz crawl hits on the server
            samantha.chapman
            samantha.chapman
            0
            2
            84

          • When I download the crawl report it downloads with multiple URLs in one box, is there a way to stop this from happening or to easily separate these URLs once downloaded?
            JordanRailsback
            JordanRailsback
            0
            2
            97

          • Limiting my MOZ crawled pages
            KeriMorgret
            KeriMorgret
            0
            7
            396

          • Multiple email Ids we can add or not in Moz pro for email's alert?
            SamWeber
            SamWeber
            0
            2
            290

          • Moz reporting client URL not valid
            DavidLee
            DavidLee
            0
            9
            142

          • How do I change the URL for the account Moz is tracking?
            moosepondarts
            moosepondarts
            0
            4
            111

          • MOZ Crawl - Search engine blocked by robots .. but it isn't.
            ChiarynMiranda
            ChiarynMiranda
            0
            8
            1.5k

          • How do I remove 'Custom Links' from my Moz.com profile?
            GregFindley.co.uk
            GregFindley.co.uk
            0
            9
            221

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy