Websites First Crawl - Over 2 Hour Suggested Wait

[hostsurfuk]

Hello SEOMoz!

We recently signed up for a free trial and on the pro dashboard it states the following.

"To get you started quickly Roger is crawling up to 250 pages on your site. You should see these results within two hours. The full crawl will complete within 7 days."

It's been nearly 24 hours and we see no results under Crawl Diagnostics however we do under rankings.

Is this normal?

Thanks

[KTaylor]

The full crawl diagnostics and rank reports are only generated once per week. So, I would give it at least 7 days before you see full data reported in your new campaign.

[hostsurfuk]

So this means I will not get any data under crawl diagnostics for a week?

Why does it not give results for up to 250 pages as stated? The domain I selected for crawling has only 20 or so pages.

[loopyal]

Louis, I think it is just a standard message, so don't worry, I doubt it will take that long.

I just started one on a 30 page site, so I expect it will show results tomorrow.

[Cyrus-Shepard]

Hi Louis,

You are right, the first starter crawl should be complete within the first couple of hours. One reason it might take longer than normal is if the SEOmoz crawler, rogerbot, has trouble crawling your site.

I took a look inside our system and it appears the website associated with your campaign doesn't have a robots.txt file. Because the SEOmoz crawler is extra polite about crawling, whenever it doesn't find a robots.txt file, this can prevent it from crawling at all.

To fix this, create a text file called "robots.txt" and with the following information in it, and place it at the root directory of your site.

User-agent: *
Disallow:

Read more about robots.txt here: http://www.seomoz.org/learn-seo/robotstxt

Unfortunately, it may be too late to add this to fix your starter crawl. So here are some things you can try:

1. After adding the robots.txt file, try creating another (duplicate) campaign and seeing if that works.
2. Or wait until your next crawl
3. Contact the help team at help@seomoz.org

Hope this helps! Best of luck with your SEO.

[hostsurfuk]

Thanks!

Last time I checked the robots.txt was there. Anyway I have rcreated this.

Will try creating a new campaign and seeing how long it takes.

Update: worked fine after around 10 minutes crawl!