Robot.txt file issue on wordpress site.

[Mustansar]

I m facing the issue with robot.txt file on my blog.

Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.

The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."

Any suggestion to over come with this issue

[renehansen]

If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.

What is your URL?

[Martijn_Scheijbeler]

Could you post the URL to your robots.txt file or post the content of the file itself?

[Andy.Drinkwater]

It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.

Drop your URL here and we can take a look.

-Andy

[PatrickDelehanty]

Hi there

Check out your robots.txt in Wordpress:

Admin Dashboard>Settings>General>Privacy -- what is setting?

I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.

I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!

[TheeDigital]

The most likely lines in your robots.txt to cause this:

User-agent: *
Disallow: /

[evolvingSEO]

Hi

You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.

If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.

Let us know if you need more help!

[AlanMosley]

What the problem is with your site I cant say because we don't have the url.

But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.

so do you really need to block the pages?

if you really do, then use meta  noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages