The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Tools
    4. How does SEOMoz crawl sites? Does it follow the sitemap?

    How does SEOMoz crawl sites? Does it follow the sitemap?

    Moz Tools
    3 3 366
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Pikimal
      Pikimal last edited by

      I only ask because in my report it returned a few "no follow" links. Was wondering how SEOMoz operates.

      1 Reply Last reply Reply Quote 0
      • HiveDigitalInc
        HiveDigitalInc last edited by

        My guess is they spider the way Googlebot spiders - finding links on your home page, following those links, and finding more links on the subsequent pages, so on and so forth. I doubt they look at the sitemap, since it rarely reflects the true importance of pages on your site, which tend to be linked to more often.

        1 Reply Last reply Reply Quote 0
        • Abe_Schmidt
          Abe_Schmidt last edited by

          Rogerbot, our crawler, obeys the robots.txt nofollow rules associated with your site. For some reason your robots.txt file is blocking our crawler from seeing those pages. For more information on how the tool works, you can view this incredible resource: http://www.seomoz.org/help/rogerbot-crawler that explains how our crawler works in vivid detail. I hope this helps!!

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • How do a run a MOZ crawl of my site before waiting for the scheduled weekly crawl?
            Kingalan1
            Kingalan1
            0
            6
            654

          • How often does seomoz crawl the site? Can you force a crawl at a specific time ?
            stewbuch1872
            stewbuch1872
            0
            6
            188

          • Does Open Site Explorer purposefully not crawl some sites?
            KeriMorgret
            KeriMorgret
            0
            4
            281

          • SEOMOZ Crawling Our Site
            AdamThompson
            AdamThompson
            0
            7
            356

          • Only 1 page is being crawled by SEOmoz for the last 2 crawls
            KeriMorgret
            KeriMorgret
            0
            6
            863

          • How to get seomoz to re-crawl a site?
            KeriMorgret
            KeriMorgret
            0
            5
            1.1k

          • I put a crawl on my site via seomoz and has come back saying only 1 page has been crawled.It has been over a week now can anyone help?
            jaytwotwenty
            jaytwotwenty
            0
            3
            739

          • SEOMoz's Crawl Diagnostics showing an error where the Title is missing on our Sitemap.xml file?
            Hurf
            Hurf
            0
            3
            1.4k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy