Correct use for Robots.txt

[EricVallee34]

I'm in the process of building a website and am experimenting with some new pages.  I don't want search engines to  begin crawling the site yet.  I would like to add the Robot.txt on my pages that I don't want them to crawl.  If I do this, can I remove it later and get them to crawl those pages?

[krissy-cca]

Hey Eric,

If you want to create and work on pages but you don't want them indexed you can add the following to the page in the section (the pages will still be crawled):

If you want NONE of your pages to be crawled (I.E the whole website) you can add the following to your robots.txt file:

User-agent: *
Disallow: /

[EricVallee34]

Damian,

Thanks for your answer, that helps.  If I add either one of the above items to my web page, and then remove it at a later date, will the search engines crawl and rank my site (at sometime after they are removed)?  In other words, and I know this sounds stupid, but does a search engine see a Robots.txt file and never visit it again?

[CPU]

Hi Eric

The guidance above means that Google when it looks to crawl your site won't its not a message to Google telling it never to come back.

Once everything is sorted, remove whichever approach you took to block the search engines and supply a sitemap to Google via the Webmaster tools. Your site should be crawled in no time after that.

Hope this helps.

[EricVallee34]

Lewis,

Thank you for the clarification!