On Page Grader can't access my URLs

[CandymanKen]

HI-

I am trying to grade some specific pages for keywords with the on page grader but it keeps telling me "Sorry, but that URL is inaccessible. "

I can reach them via the browser and they are not https. Any thoughts?

Here is a sample:

www.bulkcandystore.com/kosher-candy

Any help is appreciated.

Ken

[ChiarynMiranda]

Hey Ken,

Thanks for writing in and sorry for the confusion. I'm afraid the server for www.bulkcandystore.com/kosher-candy is returning a 403 forbidden error for our crawler, so we are not allowed to access that page with our tools. The server would respond differently to the browser vs. our tools so I would recommend contacting the webmaster for the site and white-labelling the user-agent rogerbot so that we can access the site in the future.

I hope this helps. Please let me know if you have any other questions.

Chiaryn

[CandymanKen]

Thanks for the info. I checked with my ISP and it turns out that they automatically blocked Rogerbot because of excessive requests  (this happened Jan 15). Is there a way to crawl dely between requests longer or was that maybe a one time fluke?

Thanks

Ken

[ChiarynMiranda]

Hey Ken,

Thanks for following up. You can add a crawl delay for rogerbot in your robots.txt file to limit how quickly we make requests to the site. I would just recommend that you don't add a crawl delay higher than 10 because that can cause us to be unable to crawl the site in a reasonable amount of time to complete a full crawl.

I hope that helps!

Chiaryn