The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Getting Started
    4. How Do I Scan My New Site & Grade My Work With The Robots Turned Off? For Pre-Inspection before I launch my Site?

    How Do I Scan My New Site & Grade My Work With The Robots Turned Off? For Pre-Inspection before I launch my Site?

    Getting Started
    2 2 143
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Inframan
      Inframan last edited by

      I have a new site that has all the bots turned off so google can't index my site until I'm finished it. I've been working on this site for a couple months now optimizing and I was wondering if there was anyway I can run a preliminary scan on the site for my titles, URLs, Headers, Alt Tags and pretty much anything else that will grade my work and tell me if i did anything wrong? Can MOZ do this with the Bots turned off?

      Thanks

      1 Reply Last reply Reply Quote 0
      • DirkC
        DirkC last edited by

        You could block the site for all bots except rogerbot (

        Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot).

        Alternative would be to use a local crawler like Screaming Frog (and modify the spider settings so that it ignores the robots.txt)

        rgds,

        Dirk

        1 Reply Last reply Reply Quote 1
        • 1 / 1
        • First post
          Last post
        • Not working
          eli.myers
          eli.myers
          0
          2
          19

        • Our crawler was not able to access the robots.txt file on your site
          DigitalButter
          DigitalButter
          0
          3
          64

        • Moz site crawl doesn't work
          DrainKing
          DrainKing
          0
          4
          82

        • When I crawl my site On Moz it says it can't access the robots.txt file, but crawl is fine on SEM Rush - Anyone know any reason for this?
          meghanpahinui
          meghanpahinui
          0
          3
          60

        • Site with 2 domains - 1 domain SEO opimised & 1 is not. How best to handle crawlers?
          evolvingSEO
          evolvingSEO
          0
          3
          77

        • Moz was unable to crawl your site - robots.txt
          JRGRDZ
          JRGRDZ
          0
          5
          386

        • How to scan how many pages are scanned?
          ChiarynMiranda
          ChiarynMiranda
          0
          2
          84

        • How do get Moz to spider a Development site PRE LAUNCH?
          MattRoney
          MattRoney
          0
          3
          146

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy