Crawl started on the 18th of December and still hasn't completed!

[cttgroup]

As the title states I started a crawl over 2 weeks ago now and it still hasn't completed. Does anyone know why this could be?

Thanks

[TommyTan]

Hi Laurence,

What do you mean the crawl still hasn't be completed?  Do you mean not all your sites are indexed?

I suggest you checking up on your Google Webmaster Tools and see how your site is doing.  Once you log in, click on your website then click on  the Health tab on the left hand side.  You can see all your crawl stats and errors there.  Furthermore, you can use the "Fetch as Google" tool and ask Google to crawl your pages again.

Another thing is that if you have a lot of pages and your pages are not interlinking correctly, it may take weeks for Google to crawl all your pages.  There is not set time limit on how long it takes.  Maybe submit a sitemap to speed up the process of finding all your pages?

[ChiarynMiranda]

Hi Laurance,

Thanks for writing in and sorry that your crawl never completed. I'm afraid the issue is that all crawlers are being blocked in the robots.txt file of the site and we aren't able to fully access the site to complete a crawl. If you were to update the robots.txt file to allow our crawler to access the site, we would than be able to get the crawl completed for you. You can use the user-agent rogerbot to specify our crawler or you can remove the disallow: / directive from the robots.txt file to allow us access. Once you've done that, let me know and I will have our engineers complete the crawl for you.

I look forward to hearing back soon.

-Chiaryn

[cttgroup]

Thanks I'll give that a go!