The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Tools
    4. How to set the crawler or reports to ignore

    How to set the crawler or reports to ignore

    Moz Tools
    5 3 268
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Str82u
      Str82u last edited by

      I have a mobile version of a site with a URL string that disables the mobile view on smartphones (view full site), string is like this example.com/page-name.html?mobile=off

      I need the seomoz pro reports or crawler to ignore it because the crawler visits both versions of the site then reports them as duplicate content. Is there a setting page I haven't visited yet that will set this?

      1 Reply Last reply Reply Quote 0
      • donford
        donford last edited by

        Hi Str8

        **user-agent: rogerbot
        disallow: /*mobile=**
        

        I think may do the trick... somebody may need to confirm this.

        Str82u 1 Reply Last reply Reply Quote 2
        • ChiarynMiranda
          ChiarynMiranda last edited by

          Hey Str8,

          Donford is correct. You would need to specifically block our crawler from the mobile pages in your robots.txt file. Unfortunately, we don't currently have a way to disregard specific pages or errors in the web app, but we are looking to add that function sometime in the future.

          I hope this helps. Please let me know if you have any other questions.

          Str82u 1 Reply Last reply Reply Quote 1
          • Str82u
            Str82u @donford last edited by

            Got it, thanks for jumping in so quick.

            1 Reply Last reply Reply Quote 0
            • Str82u
              Str82u @ChiarynMiranda last edited by

              Thanks for the confirmation. I hadn't wanted to put that in the robots file because if inquisitive, non-robot traffic if you get my meaning, but in it goes; can't stand seeing errors.

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              • First post
                Last post
              • Crawlers reporting upper case letter url versions although these have been 301'd to lower case !?
                Dan-Lawrence
                Dan-Lawrence
                0
                6
                152

              • Sample "toxic" links reported by google WMT not shown in OSE link report
                KeriMorgret
                KeriMorgret
                0
                2
                160

              • No (seomoz) crawler report since 7th may !!
                DavidEichholtzer
                DavidEichholtzer
                0
                8
                218

              • Why have I stopped receiving emails about crawl reports and rankings reports?
                Fitsensesports
                Fitsensesports
                0
                6
                498

              • Setting to import into gDocs an Opensitexplorer CSV report.
                Cornel_Ilea
                Cornel_Ilea
                0
                4
                671

              • Keyword Difficulty Report vs Rankings Report
                lilactree
                lilactree
                1
                3
                595

              • Crawler reporting incorrect URLs, resulting in false errors...
                AaronWheeler
                AaronWheeler
                1
                3
                646

              • 0 values reported in anchor text distribution report
                KeriMorgret
                KeriMorgret
                0
                4
                737

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy