Robots txt
-
We have a development site that we want google and other bots to stay out of but we want roger to have access.
Currently our robots.txt looks like this:
User-agent: *
Disallow: /cgi-bin/
Disallow: /development/What would i need to addd or change to let him through?
Thank you.
-
<code>This should work fine:</code><code>User-agent: *
Disallow: /cgi-bin/
Disallow: /development/</code><code>User-agent: rogerbot Allow: /development/</code> ``` -
Thank you very much.
Its looks so simple but when you've not done this before it can be scary.
I also didn't want to get it wrong and miss roger on his crawl tomorrow!