Google insists robots.txt is blocking... but it isn't.

[ahockley]

I recently launched a new website.  During development, I'd enabled the option in WordPress to prevent search engines from indexing the site.

When the site went public (over 24 hours ago), I cleared that option. At that point, I added a specific robots.txt file that only disallowed a couple directories of files.  You can view the robots.txt at http://photogeardeals.com/robots.txt

Google (via Webmaster tools) is insisting that my robots.txt file contains a "Disallow: /" on line 2 and that it's preventing Google from indexing the site and preventing me from submitting a sitemap.  These errors are showing both in the sitemap section of Webmaster tools as well as the Blocked URLs section.

Bing's webmaster tools are  able to read the site and sitemap just fine.

Any idea why Google insists I'm disallowing everything even after telling it to re-fetch?

[danatanseo]

Hi Aaron,

I identify with your frustration, but want to lead my response with the caveat that I am not a developer so there may be people here with much more technical SEO expertise than me who might have a better answer.

What I do know id that Google Webmaster Tools data is not real time and can often take days to weeks to update. It could be that the reason GWT is showing something different about your robots.txt file is because it's old information that hasn't updated yet.

When I looked at your robots.txt file, I found two sitemaps, one with 2 URLs and one with 8 URLs. This is pretty tiny. Even in the old days, conventional wisdom was that it took at least 20 content pages in order for Google to take note and index the site.

Have you tried posting the URLs of your new site on Google+? I have heard that this is a great indexing tool in addition to the Fetch as Googlebot in GWT. Just a thought!

You know, there was a time when it took 6-8 weeks for a new site to get indexed. Google has definitely sped up to the point where I think we are all expecting instant results and sometimes that just doesn't happen.

I think this just might be a matter of patience. However,  I am always willing to admit that I could be wrong and am interested to know what others think!

Dana

[smarties954]

24 hours is a short time and probably google did not reindex or even looked at your new robot.txt

Webmaster tools is way slower than bing tools, so be patient.

As a rule of thumb, I wait at least a week with google before worrying (my 2 cents)

[Christy-Correll]

Hi Aaron - You have a couple of solid answers here. Has your issue been resolved in GWT?