The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Other Research Tools
    4. Blocked Production Site from Search Engines - How to get it Crawled by Moz Crawler

    Blocked Production Site from Search Engines - How to get it Crawled by Moz Crawler

    Other Research Tools
    1 1 204
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Modi
      Modi last edited by

      I have an 'under development' site hosted, (which is an exact replica of live site as working on to add new functionalities & modules) - but its password protected, excluded from robots.txt (Disallow) & also marked noindex on all pages in the index - so that Googlebot & other Search Engines can not crawl the site

      At present the development work is almost 95% completed., Now - feel like to crawl the site through SEOMOZ Roger Bot - to know the errors and all indexed urls by Rogerbot.

      What's the best way to get Moz Bot crawl the site - but simultaneously continue it blocking its access to Search Engines

      I have gone through - https://support.google.com/webmasters/answer/93708?hl=en, it says

      a) Save it in a password-protected directory. Googlebot and other spiders won't be able to access the content-  But this way Moz will also not be able to crawl the site

      b) Use a robots.txt to control access to files and directories on your server - However it also says - It's important to note that even if you use a robots.txt file to block spiders from crawling content on your site, Google could discover it in other ways and add it to our index.

      c) Use a noindex meta tag to prevent content from appearing in our search results - It also says that a link to the page can still appear in their search results. Because we have to crawl your page in order to see the noindex tag, there's a small chance that Googlebot won't see and respect the noindex meta tag

      Password Protected thus seems the best way to continue blocking. However, continuing with it will also block Moz bot to crawl the site. Any suggestions

      Thanks

      1 Reply Last reply Reply Quote 0
      • 1 / 1
      • First post
        Last post
      • Moz Pro OnDemand Crawl fail on on WordPress site
        Natalie-Alexis
        Natalie-Alexis
        0
        2
        25

      • Why is Moz Crawling More Pages Than My Site Actually Has?
        tawnycase
        tawnycase
        0
        2
        60

      • Site Crawl - MOz Pro
        moz_support
        moz_support
        0
        2
        62

      • Has anyone had to deal with Moz crawl issues on their Zendesk support site?
        kevin.loesken
        kevin.loesken
        0
        2
        142

      • Does anyone else have issues with Moz's keyword search volume tool for Google's search engine?
        ShokIdeaGroup
        ShokIdeaGroup
        1
        5
        132

      • MOZ crawl test is not reporting on all the pages on my site.
        Abe_Schmidt
        Abe_Schmidt
        0
        2
        101

      • Moz not crawling opencart product pages
        RuthBurrReedy
        RuthBurrReedy
        0
        6
        432

      • For some reason MOZ is not indexing anything but the home page of my site. Google appears to be indexing the whole site. But it makes my MOZ reports useless. Any recommendations?
        LynnPatchett
        LynnPatchett
        0
        6
        143

      Get started with Moz Pro!

      Unlock the power of advanced SEO tools and data-driven insights.

      Start my free trial
      Products
      • Moz Pro
      • Moz Local
      • Moz API
      • Moz Data
      • STAT
      • Product Updates
      Moz Solutions
      • SMB Solutions
      • Agency Solutions
      • Enterprise Solutions
      • Digital Marketers
      Free SEO Tools
      • Domain Authority Checker
      • Link Explorer
      • Keyword Explorer
      • Competitive Research
      • Brand Authority Checker
      • Local Citation Checker
      • MozBar Extension
      • MozCast
      Resources
      • Blog
      • SEO Learning Center
      • Help Hub
      • Beginner's Guide to SEO
      • How-to Guides
      • Moz Academy
      • API Docs
      About Moz
      • About
      • Team
      • Careers
      • Contact
      Why Moz
      • Case Studies
      • Testimonials
      Get Involved
      • Become an Affiliate
      • MozCon
      • Webinars
      • Practical Marketer Series
      • MozPod
      Connect with us

      Contact the Help team

      Moz logo
      © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
      • Accessibility
      • Terms of Use
      • Privacy