The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Other Research Tools
    4. Blocked Production Site from Search Engines - How to get it Crawled by Moz Crawler

    Blocked Production Site from Search Engines - How to get it Crawled by Moz Crawler

    Other Research Tools
    1 1 204
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Modi
      Modi last edited by

      I have an 'under development' site hosted, (which is an exact replica of live site as working on to add new functionalities & modules) - but its password protected, excluded from robots.txt (Disallow) & also marked noindex on all pages in the index - so that Googlebot & other Search Engines can not crawl the site

      At present the development work is almost 95% completed., Now - feel like to crawl the site through SEOMOZ Roger Bot - to know the errors and all indexed urls by Rogerbot.

      What's the best way to get Moz Bot crawl the site - but simultaneously continue it blocking its access to Search Engines

      I have gone through - https://support.google.com/webmasters/answer/93708?hl=en, it says

      a) Save it in a password-protected directory. Googlebot and other spiders won't be able to access the content-  But this way Moz will also not be able to crawl the site

      b) Use a robots.txt to control access to files and directories on your server - However it also says - It's important to note that even if you use a robots.txt file to block spiders from crawling content on your site, Google could discover it in other ways and add it to our index.

      c) Use a noindex meta tag to prevent content from appearing in our search results - It also says that a link to the page can still appear in their search results. Because we have to crawl your page in order to see the noindex tag, there's a small chance that Googlebot won't see and respect the noindex meta tag

      Password Protected thus seems the best way to continue blocking. However, continuing with it will also block Moz bot to crawl the site. Any suggestions

      Thanks

      1 Reply Last reply Reply Quote 0
      • 1 / 1
      • First post
        Last post
      • Moz Pro OnDemand Crawl fail on on WordPress site
        Natalie-Alexis
        Natalie-Alexis
        0
        2
        25

      • In the Moz Site Crawl, what does "External Links" mean?
        NickW816
        NickW816
        1
        2
        180

      • Moz Site Crawl Test 404
        Dr-Pete
        Dr-Pete
        0
        4
        99

      • Has anyone had to deal with Moz crawl issues on their Zendesk support site?
        kevin.loesken
        kevin.loesken
        0
        2
        142

      • Rogerbot will not crawl my site! Site URL is https but keep getting and error that homepage (http) can not be accessed. I set up a second campaign to alter the target url to the newer https version but still getting the same error! What can I do?
        DirkC
        DirkC
        0
        3
        200

      • Does anyone else have issues with Moz's keyword search volume tool for Google's search engine?
        ShokIdeaGroup
        ShokIdeaGroup
        1
        5
        132

      • MOZ crawl test is not reporting on all the pages on my site.
        Abe_Schmidt
        Abe_Schmidt
        0
        2
        101

      • How do I get Moz to crawl my site?
        Modi
        Modi
        0
        3
        1.8k

      Get started with Moz Pro!

      Unlock the power of advanced SEO tools and data-driven insights.

      Start my free trial
      Products
      • Moz Pro
      • Moz Local
      • Moz API
      • Moz Data
      • STAT
      • Product Updates
      Moz Solutions
      • SMB Solutions
      • Agency Solutions
      • Enterprise Solutions
      • Digital Marketers
      Free SEO Tools
      • Domain Authority Checker
      • Link Explorer
      • Keyword Explorer
      • Competitive Research
      • Brand Authority Checker
      • Local Citation Checker
      • MozBar Extension
      • MozCast
      Resources
      • Blog
      • SEO Learning Center
      • Help Hub
      • Beginner's Guide to SEO
      • How-to Guides
      • Moz Academy
      • API Docs
      About Moz
      • About
      • Team
      • Careers
      • Contact
      Why Moz
      • Case Studies
      • Testimonials
      Get Involved
      • Become an Affiliate
      • MozCon
      • Webinars
      • Practical Marketer Series
      • MozPod
      Connect with us

      Contact the Help team

      Join our newsletter
      Moz logo
      © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
      • Accessibility
      • Terms of Use
      • Privacy