On Page Optimization, F on all keywords?

[stokesautogroup]

This is our robots.txt file, everything looks normal. Now that you say that, I do notice in our dashboard we have a high amount of crawl issues. At least 1 on each of our campaigns.

User-agent: *
Disallow: /request-more-info/
Disallow: /test-drive-this-car/
Disallow: /confirm-availability/
Disallow: /request-vehicle-quote/
Disallow: /send-to-mobile/
Disallow: /email-a-friend/
Disallow: /gallery/
Disallow: /eprice/
Disallow: /print-auto/
Disallow: /autocheck/
Disallow: /print-coupons/
Disallow: /request-video-walkaround/
Disallow: /get-e-finance/
Disallow: /condition-report/
Crawl-delay: 2
Sitemap: http://www.stokesbrowntoyotahiltonhead.com/resrc/xmlsitemap/xml-sitemaps/

User-agent: MJ12bot
Disallow: /
User-agent: 008
Disallow: /
User-agent: DTS Agent
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: crawler4j
Disallow: /
User-agent: Goodzer
Disallow: /

[Andy.Drinkwater]

I get a 403 for every page in the site. Is there something with your host that is blocking crawlers?

Edit-- OK , so it isn't something with your host as I tried to crawl another site on the same server, and that works without issue.

-Andy

[Linda-Vassily]

Even the robots.txt page is 403...

[Andy.Drinkwater]

I suspect this is either a permissions problem in the site structure or an issue in .htaccess, neither which we can see.

Aside from some messy code, issues on the cosole log and iframes on the page, I can't see that these would cause the problem because the pages are being cralwed and indexed.

-Andy

[stokesautogroup]

We are also receiving these issues as well. Seems like its a hit or miss with the crawling.

9T3sDlq.png

[stokesautogroup]

Any ideas of where we should start to narrow down this issue?

[Andy.Drinkwater]

I would start with contacting the developers and ask them if they have any clue about it Daniel. Failing that, without having a good root around, it's hard to say at a glance what is going on. There certainly are a lot of more general site issues.

-Andy

[Linda-Vassily]

Have you ever been able to see a non-all F report from Moz? If so, I would see what has changed since then. But yes, your developers should be able to give you the answers you are looking for.

[stokesautogroup]

All F's.

Looks like our site devs have fixed the issue. I just ran the page grader and we received an A. I guess we wait until Moz crawls our site again to see our updated info.

[stokesautogroup]

Looks like our host solved the problem. They mentioned they were blocking the rogerbot crawler. Anyone have an idea of how long we have to wait until the page grades and other info updates in our dashboard?