Robots.txt blocking Metadata description being read

[IMM]

I have updated my Robots.txt file recently to fix the description snip-it on Google reading "A description for this result is not available because of this site's robots.txt – learn more".  How can I have this updated to read my META data?

[matbennett]

If you deny the robot access to the URL then it will not be able to read the meta.  There isn't a way around this - as that is the purpose of denying robots.    If the robot is allowed then it will update when googlebot recrawls your page and updates its index.  How long that takes varies from site to site and page to page.

[IMM]

I only have pages in the checkout flow blocked.  My homepage and other pages should be accessible.

[CleverPhD]

It sounds like you have something going on like what Matt Cutts talks about here

http://www.youtube.com/watch?v=KBdEwpRQRD0

You have a result showing up in the SERPs, even though it is in robots.txt.   Basically, the reason it is still in the SERPs is because other pages are linking to the URL on your page.

I am going to assume that you want to keep these pages out of the index.

As you already have pages in the index, you need to get them removed, not just block them.  I would suggest using  a noindex meta tag and then letting the crawler crawl the page.  The robots.txt stops the bot cold and does not let it read anything else.  It does not let the bot read the meta tag.  If you let it read the noindex meta tag that tag directs Google to take the page out of the search results.

https://support.google.com/webmasters/answer/93708?hl=en

"When we see a noindex meta tag on a page, Google will completely drop the page from our search results, even if other pages link to it."

That said, if you have made a mistake and have been blocking Google when you did not mean to.  Make sure that you do not use the noindex meta tag on those pages, and make sure you are not blocking Google in your robots.txt.  If that is the case and you are still seeing the wrong info in the SERPs, you do need to wait a little while.  The updates will not be instantaneous and may take a few weeks before being updated in the SERPs.   In the mean time, just double check that everything in your robots.txt is correct etc.

Be patient and good luck!

[bridget.randolph]

Hi IMM,

What page are you trying to fix? And what did you change in the robots.txt file to fix it?