The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Pro
    4. Seomoz crawling filtered pages

    Seomoz crawling filtered pages

    Moz Pro
    5 2 394
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • nvs.nim
      nvs.nim last edited by

      Hi,

      I just checked an seo campaign we started last week, so I opened seomoz to see the crawl diagnostics.

      Lot's of duplicate content & duplicate titles showing up, but that's because Rogerbot is crawling all of the filtered pages as well. How do I exclude these pages from being crawled?

      /product/brand-x/3969?order=brand&sortorder=ASC
      /product/brand-x/3969?order=popular&sortorder=ASC
      /product/brand-x/3969?order=popular&sortorder=DESC&page=10
      /product/brand-x/3969?order=popular&sortorder=DESC&page=11

      1 Reply Last reply Reply Quote 0
      • My-Favourite-Holiday-Cottages
        My-Favourite-Holiday-Cottages last edited by

        Last I checked, Rodger obays robot.txt and meta-robot commands as he trys to simulate what google would crawl. If he can crawl those pages, google probably is as well.

        I think if you rel-canonical them properly, or no-idex them properly e.t.c. It should show as a normal page.

        nvs.nim 1 Reply Last reply Reply Quote 0
        • nvs.nim
          nvs.nim @My-Favourite-Holiday-Cottages last edited by

          Rel-canonical will be an immense job, would it be ok using robots.txt like this

          disallow: /?order=

          disallow: /?sortorder=

          ...

          ?

          My-Favourite-Holiday-Cottages nvs.nim 2 Replies Last reply Reply Quote 0
          • My-Favourite-Holiday-Cottages
            My-Favourite-Holiday-Cottages @nvs.nim last edited by

            I'd rel-canonical if you can, as theres still nothing stopping links to them being indexed. It might stop Rodger/Google from crawling them, but the potential indexation issues won't go away. Otherwise perhaps no-index them.

            I'd usualy go as far to do re-prev and rel-next for the paginated searches as well.

            1 Reply Last reply Reply Quote 1
            • nvs.nim
              nvs.nim @nvs.nim last edited by

              So if the site has a structure like this:

              www.xyz.com/overview

              and the filter on this page has several options like /?order= , ?brand=, .... I have to rel-canonical them al to www.xyz.com/overview

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              • First post
                Last post
              • Unable to crawl pages
                KingPings
                KingPings
                0
                5
                122

              • Crawled pages are missing and showing just 1 page crawled
                KeriMorgret
                KeriMorgret
                0
                3
                296

              • Why did SEOMoz only crawl 1 page?
                KeriMorgret
                KeriMorgret
                0
                5
                439

              • Does the SEOMoz weekly crawl that highlights no meta description tag, take into account if there is a meta robots noindex,follow tag on the pages it indicates the missing meta descriptions?
                ShaMenz
                ShaMenz
                0
                4
                649

              • Crawling One Page
                KeriMorgret
                KeriMorgret
                0
                2
                654

              • SEOMoz Pro still hasn't crawled 10k pages for one campaign
                KeriMorgret
                KeriMorgret
                0
                2
                685

              • SEOmoz crawl diagnostics report - what are the duplicate pages urls?
                SteveOllington
                SteveOllington
                0
                4
                959

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy