How to fix and issue with robot.txt ?

[skehoe]

I am receiving the following error message through webmaster tools

http://www.sourcemarketingdirect.com/: Googlebot can't access your site

Oct 26, 2012
Over the last 24 hours, Googlebot encountered 35 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.

The site has dropped out of Google search.

[Matt-Williamson]

have you made sure your robots.txt is loading in your browser by adding robots.txt after your domain same as a normal page and can you see contents? has your site been down in this period? have you changed the contents of the file just before this issue? are you sure googlebot hasnt come back since that date - whats your analytics say? do an index site: search for your domain to see if it is in google.

[skehoe]

Thanks Matt.

There is no robots.txt as far as I can see. Is there a plugin I can use for wordpress?

The site was down for 2 days last month while hte original host transfered the site over to me.

Right now a site search says their are 13 pages indexed.

Just concerned that this site has always ranked number 1 for a company name search and now they are not on the first 10 pages in Google.

[Matt-Williamson]

just a quick check you have got wordpress visible to search engines set in the admin area? if not it will be set to disallow googlebot to crawl it.

it is in admin €€€- options - privacy and select appropriate box - default is no index, no follow.

[MarieHaynes]

I've seen several people ask this very same question over the last week in different forums.  I am wondering if the major outages with hurricane Sandy have affected several hosts or DNS's.

Your robots.txt looks fine to me.

I'm guessing that you will completely recover once Google has a chance to fully crawl the site again.

[skehoe]

Thanks for your answer.

I have received this message from Google

**http://www.sourcemarketingdirect.com/ **using the Meta tag method (less than a minute ago). Your site's home page returns a status of 500 (Internal server error) instead of 200 (OK)

It looks like the permalink structure has changed but I'm not sure how.

[Matt-Williamson]

Hi Stacey

What plugins do you have running - any caching plugins such the W3 Total Cache plugin?

Are you able to access your servers error logs to see if you can see anything there?