How do get Moz to spider a Development site PRE LAUNCH?

[bjs2010]

Hi,

Does anyone know how we could get Moz to browse a development site before launch? But without Google and other engines indexing it?

Thanks

[Keszi]

Well, you could block Google and Bing from crawling your website (via Robots.txt) and run a Crawl via Moz, that's possible I think.

Istvan

[MattRoney]

That would do it! Try this in robots.txt:

User-agent: Rogerbot
Disallow:

User-agent: *
Disallow: /

Keep in mind that this will only block crawlers that obey robots.txt. Once Roger crawls your site, it may be a good idea to lock the dev site down entirely.