Screaming Frog, Moz and other crawlers

[mccormackmorrison]

Hi

Ignorant question, but is it possible to use Screaming Frog or the Moz crawler or any other reputable crawler for a site still in development i.e. it is yet to be indexed? If so, could someone provide some quick instructions on how this can be done.

Thanks in advance for any support.

Neil

[ssiebn7]

If the site is online or on a domain or somewhere where Screaming Frog can reach it you can spider it with the software yes.

But it has to be online somewhere so the software can reach it.

Did this answer your question?

[Chris.Menke]

Tim,

As long as you can get to the site remotely via a website address you should be good to go. However, if the site is blocking crawlers via robots.txt file or meta robots tag rogerbot won't access it.  On the other hand, screamingfrog has a setting to tell it to ignore the robots.txt file if one exists.

[mccormackmorrison]

Yes, thanks for the answer.

[mccormackmorrison]

Thanks for the answer, much appreciated.