Why wont rogerbot crawl my page?

[friendoffood]

How can I find out why rogerbot won't crawl an individual page I give it to crawl for page-grader?  Google, bing, yahoo all crawl pages just fine, but I put in one of the internal pages fo page-grader to check for keywords and it gave me an F -- it isn't crawling the page because the keyword IS in the title and it says it isn't.   How do I diagnose the problem?

[Martijn_Scheijbeler]

Hi Theodore, could you share the specific URL with us so we could help you diagnose what the issue could be?

[donford]

Hi Theodore,

Last time I looked at this issue for another community member they had a site that had huge images and slow script. This decreased the load time of the page and Roger just got frustrated. Rogerbot is not as sophisticated as the huge Search Engines crawlers and can easily be put off.

As Martijn asked, for us to help we really would have to look at the site to pick out possible issues.

[friendoffood]

Hi sure, thanks.  This page shouldn't have a speed issue but maybe you can see what the issue is:

www.qjamba.com/local-coupons/wentzville/mo/all

Thanks.

[donford]

I know in robots.txt any URL's are case sensitive, I am not sure about user agents (bots/crawlers) but you do have RogerBot spelled with a capitol "B", changing it to lower case (Rogerbot) may fix the issue.

Another thing to test would be to simply remove the mass exclusion just to see if Rogerbot somehow is being blocked by it. Let me know how it goes.

User-agent: *
Disallow: /

[friendoffood]

Thanks.  The robots.txt file was the problem.  It originally (yesterday) excluded rogerbot (by default) and then I remembered that and put it in as rogerbot but that didn't work.  So I changed it to RogerBot  and that didn't work.  Today I removed the robots.txt file completely and it worked.  Then I put it back with rogerbot and it is working.

It APPEARS that maybe it read the robots.txt  yesterday before i put in rogerbot and for some reason didn't read it after I put it in.  Will never know but it is now working.

Thanks for the help!

[donford]

Very glad to see you got it working!

You can mark the question as answered to let others know it is fixed.