Robots.txt unblock

[Elchanan]

I'm currently having trouble with what appears to be a cached version of robots.txt. I'm being told via errors in my Google sitemap account that I'm denying Googlebot access to the entire site. I uploaded clean and "Allow" robots.txt yesterday, but receive the same error.

I've tried "Fetch as Googlebot" on the index and other pages, but still the error. Here is the latest:

| Denied by robots.txt |
| 11/9/11 10:56 AM |

As I said, there in not blocking on the robots.txt for 24 hours.

HELP!

[RyanKent]

We cannot view your link as it is to your private Google WMT account.

Log into your Google WMT account then choose Site Configuration > Crawler Access. The screen will show how many hours or days it has been since your robots.txt file has been last updated.

Also, visit your site's robots.txt file to ensure it appears accurate. Most sites share their robots.txt file at www.mysite.com/robots.txt

Generally speaking, you want the cleanest robots.txt file possible. The robots.txt file should be the absolute last method to block crawler access in cases where no other method can be implemented. Many site owners do more harm then good with their settings.

If more assistance is needed please share the URL to your robots.txt file along with the URL to a page which is being inappropriately blocked.

[Elchanan]

Thank You Ryan.

It all started with human mistake. The webmaster copied a stage server into live server include "Disallow /" command

The robots text he uploaded yesterday, is the original. Should be okay.

Configuration > Crawler Access says

Downloaded	Status	Home page access
thesite	Nov 8, 2011	200 (Success)	Googlebot is blocked from thesite

I've reading that it can take 180 days(!) to unblock it. Is that right?

[RyanKent]

You are showing the status from yesterday's date, Nov 8th. It seems Google has not attempted to crawl your site today. The next time it does attempt to crawl your site it will notice the changed robots.txt file and can then crawl normally.

I've reading that it can take 180 days(!) to unblock it. Is that right?

No. Google crawls much more frequently. Your site will likely be crawled again within the next day or so. Try checking again tomorrow.

[Elchanan]

OK. I tried now the "Configuration > Crawler Access" but it still show Nov 8th.

Then, I took the risk, and clicked on "Fetch as Googlebot" - Taddam, I'm okay

I also able to resubmit the sitemaps XML.