Crawl Diagnostic | Starter Crawl taken 14hrs.. so far

[RoryMacDonald]

We started a starter crawl 14hrs ago and it's still going, can anyone help on why this is taking so long, when it says '2 hrs' on the interface..

Thanks,

Rory

[avassa]

Rory,

I would guess that this crawl session has hung-up; it would be a good idea to start a new session. The session could have been left in the middle due to a server side issue on your website or a temporary drop in connection between the API server and your website's server.

[RoryMacDonald]

How would I reset the crawl? I don't appear to have an option to?

[avassa]

Rory,

What is the sub-domain that you are crawling? It may just be that there is a lot of content to crawl.

[avassa]

And here is how you reset the crawl:

1. On your webserver, edit the robots.txt file.

2. Block the seomoz bot from crawling the site by blocking its access to the root.

You can do so by adding the following lines:

User-agent: rogerbot

Disallow: /

This would end the crawl session.

But, before you do this, it may a good idea to check if your site indeed has a lot of content and outgoing links?

[RoryMacDonald]

It's a fairly big site, but it does say:

'To get you started quickly Roger is crawling up to 250 pages on your site. You should see these results within two hours. The full crawl will complete within 7 days.'

There's no option to do anything else, like cancel, reset etc — it just says 'Starter crawl in progress', it's been 16hrs now + bit frustraing as needed to send this through to a client this morning.. Anyone from SeoMoz around to look into this?

[avassa]

Send an email to help (at) seomoz.org for someone to have a look.

[RoryMacDonald]

I've asked — now heard yet, think i'll wait to hear.

Thanks for your help, appreciate it.

[KeriMorgret]

Hi Rory. Most of our help desk is on holiday today, since it's the Fourth of July in the states. We do have a record of your ticket and one other person who is having a slow starter crawl, and a help desk specialist is looking into this now. Sorry for the delays.

Keri