The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Other Questions
    4. Moz Crawler not working on beta site

    Moz Crawler not working on beta site

    Other Questions
    4 2 127
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Ideas-Money-Art
      Ideas-Money-Art last edited by

      Hello Mozzers,

      We are launching a new site and before we do we wanted to run a crawl but unfortunately no diagnosis came back. The beta site is not password protected. It does have noindex,nofollow but it shouldn't matter. So I am not sure why nothing has been detected?

      Thanks!

      i8ZEoCT.png

      1 Reply Last reply Reply Quote 0
      • LesleyPaone
        LesleyPaone last edited by

        If it does have a no index no follow, then the crawler acted as expected, right? It didn't index the pages or crawl them.

        1 Reply Last reply Reply Quote 3
        • Ideas-Money-Art
          Ideas-Money-Art last edited by

          I am confused because nofollow and noindex have always been detectable in the past with live sites. Is this new?

          How am I suppose to go around this since I don't want to launch a beta site with tons of errors which could hurt the SEO?

          1 Reply Last reply Reply Quote 0
          • Ideas-Money-Art
            Ideas-Money-Art last edited by

            Hi everyone!

            I wanted to follow up with everyone who experienced the same issue. Unfortunately the moz crawler doesn't crawl any noindex,nofollow which proves its limits. There is another tool called screamingfrog that will allow you to crawl all the noindex, nofollow pages. This tool is literally saving me as I was able to detect many errors on the Beta Site. Without it we would have launched the new site and our SEO would have taken a major hit.

            Thanks!

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • Can I limit Moz crawl to site map?
              bradsimonis
              bradsimonis
              0
              7
              43

            • Moz unable to crawl site
              Alaeddin
              Alaeddin
              1
              3
              84

            • Moz was unable to crawl your site on Sep 18, 2018
              Netvilox
              Netvilox
              0
              4
              159

            • Does the Moz crawler ignore the robots.txt file?
              eli.myers
              eli.myers
              0
              2
              264

            • Moz PRO vs Moz Analytics value and differences?
              DavidLee
              DavidLee
              0
              2
              640

            • "Finish" button for creating a new campaign in Moz Analytics does not work...
              mlapierre
              mlapierre
              0
              6
              203

            • Moz Hasn't Crawled my Sites In Two Weeks
              AaronWheeler
              AaronWheeler
              0
              3
              157

            • What is wrong with Moz?
              JoelDay
              JoelDay
              0
              4
              169

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy