The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Tools
    4. Crawl Errors from URL Parameter

    Crawl Errors from URL Parameter

    Moz Tools
    2 2 275
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • WrightIMC
      WrightIMC last edited by

      Hello,

      I am having this issue within SEOmoz's Crawl Diagnosis report.  There are a lot of crawl errors happening with pages associated with /login.

      I will see site.com/login?r=http://.... and have several duplicate content issues associated with those urls.

      Seeing this, I checked WMT to see if the Google crawler was showing this error as well.  It wasn't.

      So what I ended doing was going to the robots.txt and disallowing rogerbot.

      It looks like this:

      User-agent: rogerbot

      Disallow:/login

      However, SEOmoz has crawled again and it still picking up on those URLs.   Any ideas on how to fix?

      Thanks!

      1 Reply Last reply Reply Quote 0
      • JoelDay
        JoelDay last edited by

        Hi Tony,

        I need more information from you in order to check this out. I'm going to send you a support ticket to reply to.

        Thanks,
        Joel.

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        • Does anyone know the linking of hashtags on Wix sites does it negatively or postively impact SEO. It is coming up as an error in site crawls 'Pages with 404 errors' Anyone got any experience please?
          Nozzle
          Nozzle
          0
          5
          82

        • Site Crawl 4xx Errors?
          mostcg
          mostcg
          0
          3
          189

        • Moz Crawl Report more urls?
          MattRoney
          MattRoney
          0
          2
          240

        • Site Crawl Error
          nopsts
          nopsts
          0
          5
          434

        • Rogerbot crawls my site and causes error as it uses urls that don't exist
          AdamWormann
          AdamWormann
          0
          2
          1.2k

        • 5XX (Server Error) on all urls
          Martijn_Scheijbeler
          Martijn_Scheijbeler
          0
          4
          4.9k

        • Seo moz has only crawled 2 pages of my site. Ive been notified of a 403 error and need an answer as to why my pages are not being crawled?
          nitro-digital
          nitro-digital
          0
          9
          319

        • How to handle crawl diagnostic errors for the same url. /products & /products/
          Keszi
          Keszi
          0
          2
          678

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy