Welcome to the Q&A Forum

Hi, Usin; User-agent: * Disallow: /folder/subfolder is fine, however if you have information stored in your website that you certainly want crawled make sure it is in your site map and use ... User-agent: * allow: /folder/subfolder adding a no follow attribute to all of your pages wont be practical, if a spam crawler ignores the robots.txt it will ignore your no follow attribute. If anything new occurs with robots.txt check large website's robots.txt as they always update to new trends i.e www.google.com/robots.txt Hope this helps:)