The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Tools
    4. Crawl Errors from URL Parameter

    Crawl Errors from URL Parameter

    Moz Tools
    2 2 275
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • WrightIMC
      WrightIMC last edited by

      Hello,

      I am having this issue within SEOmoz's Crawl Diagnosis report.  There are a lot of crawl errors happening with pages associated with /login.

      I will see site.com/login?r=http://.... and have several duplicate content issues associated with those urls.

      Seeing this, I checked WMT to see if the Google crawler was showing this error as well.  It wasn't.

      So what I ended doing was going to the robots.txt and disallowing rogerbot.

      It looks like this:

      User-agent: rogerbot

      Disallow:/login

      However, SEOmoz has crawled again and it still picking up on those URLs.   Any ideas on how to fix?

      Thanks!

      1 Reply Last reply Reply Quote 0
      • JoelDay
        JoelDay last edited by

        Hi Tony,

        I need more information from you in order to check this out. I'm going to send you a support ticket to reply to.

        Thanks,
        Joel.

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        • Site Crawl 4xx Errors?
          mostcg
          mostcg
          0
          3
          189

        • Moz Crawl Test error
          donford
          donford
          0
          2
          363

        • Site Crawl Error
          nopsts
          nopsts
          0
          5
          434

        • Rogerbot crawls my site and causes error as it uses urls that don't exist
          AdamWormann
          AdamWormann
          0
          2
          1.2k

        • SEOmoz showing crawl errors but webmastertools says no errors, need help!
          bobsnowzell
          bobsnowzell
          0
          5
          399

        • Crawl test tool from SEOmoz - which URLs does it actually crawl?
          ShaMenz
          ShaMenz
          0
          4
          787

        • Crawl Diagnostic Errors
          rosstaylor
          rosstaylor
          0
          9
          923

        • Errors on my Crawl Diagnostics
          rhutchings
          rhutchings
          0
          2
          645

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy