The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Link Explorer
    4. Moz crawling bot

    Moz crawling bot

    Link Explorer
    3 2 3.7k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • superseopl
      superseopl last edited by

      Hi guys,

      in OpenSiteExplorer -> Top Pages, there are no page titles displayed in a raport for certain domain, and "HTTP Status" column shows: "Blocked by robots.txt".

      I tried to find out what the ID of Moz crawling bot is, and on this page: http://moz.com/community/q/seomoz-spider-bot-details someone says it's: Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot).

      However, my robots.txt doesn't have such entry. Take a look:

      Automatically banned scanners and crawlers section

      User-agent: 008

      Disallow: /

      user-agent: AhrefsBot

      Disallow: /

      User-agent: MJ12bot

      Disallow: /

      User-agent: metajobbot

      Disallow: /

      User-agent: Exabot

      Disallow: /

      User-agent: Ezooms

      Disallow: /

      User-agent: fyberspider

      Disallow: /

      User-agent: dotbot

      Disallow: /

      User-agent: MojeekBot

      Disallow: /

      Section end

      What could be the problem here, then? Why does the Moz bot think I'm blocking it?

      1 Reply Last reply Reply Quote 0
      • KeriMorgret
        KeriMorgret last edited by

        If you pull out dotbot from your exclusions, you should be fine.

        http://www.opensiteexplorer.org/dotbot

        superseopl 1 Reply Last reply Reply Quote 2
        • superseopl
          superseopl @KeriMorgret last edited by

          Thank you Keri, problem solved.

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Why moz is not crawling my Backlinks & Keyword ?
            Williamh20
            Williamh20
            1
            5
            85

          • My company has about 70 live pages. Moz crawled 6k. Why?
            effectdigital
            effectdigital
            1
            3
            73

          • Moz crawling http rather than https site
            dave.kudera
            dave.kudera
            0
            3
            91

          • Does the Moz Pro site crawl, crawl password protected sites?
            tawnycase
            tawnycase
            0
            4
            114

          • Angular SPA & MOZ Crawl Issues
            Raptor-crew
            Raptor-crew
            0
            5
            342

          • Moz Crawl Issue?
            DavidLee
            DavidLee
            0
            3
            71

          • Why is Moz not crawling my backlinks
            DavidLee
            DavidLee
            0
            6
            708

          • How Is a Page Crawled by Moz When Moz Says 'No Links'?
            xerox432
            xerox432
            0
            9
            153

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy