Some web 2.0 sites blocked by robot

[pnb567]

I had created some web2.0 sites, google also indexed those sites but those sites robots.txt are disallow, this mean Google will not give any credit on ranking ? I have seen in google, those sites meta description was written - A description of this site is not available due to the robot.txt file. I checked that file, it is written disallow. Can google still count those sites for the ranking purpose ?

[Devanur-Rafi]

Hi,

Pages blocked by robots.txt file will not be considered for ranking. You still see those pages in Google's index is because of their mentions on other websites. Google has been doing link discovery from websites other than these Web2.0 sites of yours. So to stop Google completely from indexing these pages, you should go in for page-levelrobots meta tag. Here you go for more: https://support.google.com/webmasters/answer/93710?hl=en

Once you implement the page-level robots meta tag to block those pages, Google will remove them from the index after the next crawl. You can expedite this process by using the link removal procedure in webmaster tools account.

Here you go for more: https://support.google.com/websearch/troubleshooter/3111061?rd=1

Good Luck to you my friend.

Best regards,

Devanur Rafi