Webmaster tools - Restricted by robots.txt

[powpowteam]

This post is deleted!

[STPseo]

If these are not in the robot text file I would just give Google a few days to crawl them. You can check manually if they are in the index by doing a search for the URL. You can also submit a sitemap with the URLs no longer being restricted, and again, give it a few days for Google to catch up.

In this case more important than the robot file are the robot meta commands on each page. If you have the pages tagged with:

<meta name="robots" content="index" />

You are telling Google and other search engines that the page is okay to add to their search results.

[RyanKent]

The "resticted by robots.txt" message is designed to inform you Google attempted to crawl the page but was prevented from doing so based on your robots.txt settings.

If you remove the restriction in robots.txt, then the next time Google attempts to crawl the page they will be successful.

Based on your current robots.txt file, those URLs will be crawled on the next attempt.

A bit of additional information. Your current robots.txt file shows as:

User-agent: *

Allow: /

DisAllow: /news

That should work but the proper format should be:

User-agent: *

Disallow: /news

For more information see http://www.robotstxt.org/

[nasavsisro]

I have same question my site is http://www.wrock.org/  and i update robots.txt file for block some of pages but now Google Webmaster Tool show "88 pages to URL restricted by robots.txt" i already updated robots file to default but its more than 20 day they are still their and i lost my 50% of traffic after this i think bcoz of my experiment  New file is here http://www.wrock.org/robots.txt

[RyanKent]

Hi Sandeep.

While I see it commonly used, "Allow" is not a valid syntax for robots.txt file. You can confirm it with this validation tool: http://tool.motoricerca.info/robots-checker.phtml

Since the code is not valid, I am not sure how Google will handle it. My best guess is you should be fine, but you can also use Google's robots.txt tool: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=35237&rd=1

Any changes to your robots.txt file should be picked up immediately on the next crawl. Can you please share the specific dates involved when the pages were restricted by robots.txt? There is a good chance your site is fine now and you are looking at old records.