The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Tools
    4. Rogerbot crawls my site and causes error as it uses urls that don't exist

    Rogerbot crawls my site and causes error as it uses urls that don't exist

    Moz Tools
    2 2 1.2k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • spiralsites
      spiralsites last edited by

      Whenever the rogerbot comes back to my site for a crawl it seems to want to crawl urls that dont exist and thus causes errors to be reported...

      Example:- The correct url is as follows:

      /vw-baywindow/cab_door_slide_door_tailgate_engine_lid_parts/cab_door_seals/genuine_vw_brazil_cab_door_rubber_68-79_10330/

      But it seems to want to crawl the following:

      /vw-baywindow/cab_door_slide_door_tailgate_engine_lid_parts/cab_door_seals/genuine_vw_brazil_cab_door_rubber_68-79_10330/?id=10330

      This format doesn't exist anywhere and never has so I have no idea where its getting this url format from

      The user agent details I get are as follows:

      IP ADDRESS: 107.22.107.114
      USER AGENT: rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+pr1-crawler-17@moz.com)

      1 Reply Last reply Reply Quote 0
      • AdamWormann
        AdamWormann last edited by

        The first thing I would do is download the crawl report as an excel sheet.  You can do this from your crawl report page.

        From there, sort by the 404 error column, bringing "True" to the top.  The top of the list is now the broken URL's.  One of the very last columns on the right is the "referrer" column.  This will show you the page where Roger is getting the bad link from.

        Make Sense?

        1 Reply Last reply Reply Quote 2
        • 1 / 1
        • First post
          Last post
        • Does anyone know the linking of hashtags on Wix sites does it negatively or postively impact SEO. It is coming up as an error in site crawls 'Pages with 404 errors' Anyone got any experience please?
          Nozzle
          Nozzle
          0
          5
          82

        • Moz-Specific 404 Errors Jumped with URLs that don't exist
          effectdigital
          effectdigital
          0
          18
          200

        • Spammy inbound links: Don't Fix It If It's Not Broken?
          BlueprintMarketing
          BlueprintMarketing
          0
          4
          82

        • I have 2 linking root domains on my URL. But I don't get the whole Root domain thing. So I don't understand how I can improve it?
          MissThumann
          MissThumann
          0
          3
          222

        • Pages with Temporary Redirects on pages that don't exist!
          Raptor-crew
          Raptor-crew
          0
          8
          974

        • Its been over a month, rogerbot hasn't crawled the entire website yet. Any ideas?
          BlueprintMarketing
          BlueprintMarketing
          0
          4
          322

        • Why won't scheduled crawl of my site begin?
          Prime85
          Prime85
          0
          3
          585

        • It won't let me print the secon or third pages of site errors.
          KeriMorgret
          KeriMorgret
          0
          2
          760

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy