No (seomoz) crawler report since 7th may !!

[DavidEichholtzer]

Hi Mozteam,

I added a site less than 200 pages in the tool "seomoz crawler", at May 7.

We are May 15 and the tool always displays "crawl in progress."

Do you have a problem about this tool?

This is embarrassing ...

Thank you for your reply.

David

France

[Perfect007]

SEOMoz Tool crawls a website once a week, but you can do a separate crawl of any site (up to 3000 URLs) at http://pro.seomoz.org/tools/crawl-test. They generally update crawl Thursday to Thursday along with Ranking.

[DavidEichholtzer]

Hi

thank you but it is precisely this tool I'm talking about ...

the crawl is "in progress" since May 7 ..

David

[PaddyDisplays]

you have no results at all?  as perfect007 said its updated once a week, so it normally says "in progress"

But if you still get no results by tomorrow, then report to the seomoz staff,  along time ago I had a crawl problem where it was only crawling about 500 pages of my site, when we had 10,000+ pages.   There was a technical problem with the crawl with my site which they fixed very quickly

[DavidEichholtzer]

Thank you,

no result at all... since may 7..

meanwhile I used Integrity .. but it is unfortunate that the report crawler does not work.

[Brian-H]

I've had a few sites that block the crawler (which obeys strictly the robots.txt). I've had this also with wordpress development sites that are "discourage search engines" box ticked.

Check the robots.txt and header in the page to make sure you are not blocking crawlers and if not put a ticket request in with seomoz and they will quickly look in to it.

Brian

[JoelDay]

Hey David,

Thanks for reaching out. I looked at your account and don't see any crawls that started on the 7th. I do see a crawl that started on the 3rd and hasn't completed because you are blocking all user agents in your robots.txt file.

You have this directive listed:

User-agent: *
Crawl-delay: 10
Disallow: /

That is going to prevent all bots from accessing your site, not just RogerBot.

You'll want to either remove that or add a directive before that for User-agent: RogerBot allowing full access. Once that's done, just wait for the crawl to refresh and it should complete just fine.

Thanks!
Joel.

[DavidEichholtzer]

Hi Joel

if I tell you that I am doing SEO for 13 years you will not believe me :-))
tiredness or inattention ...

thank you Joel and other contributors for your answers, I appreciate it.

David