XML Sitemap instruction in robots.txt = Worth doing?

[AshSEO2011]

Hi fellow SEO's,

Just a quick one, I was reading a few guides on Bing Webmaster tools and found that you can use the robots.txt file to point crawlers/bots to your XML sitemap (they don't look for it by default).

I was just wondering if it would be worth creating a robots.txt file purely for the purpose of pointing bots to the XML sitemap?

I've submitted it manually to Google and Bing webmaster tools but I was thinking more for the other bots (I.e. Mozbot, the SEOmoz bot?).

Any thoughts would be appreciated!

Regards,

Ash

[OptimizeSmart]

It is not going to make any difference. Time is better spend in fixing crawling & indexing issues of the website.

[john4math]

I think it's worth it as it should only take a few minutes to set up, and it's good to have a robots.txt, even if it's allowing everything.  Put a text file named "robots.txt" in your root directory with:

<code>User-agent: *
Disallow: 

Sitemap: http://www.yourdomain.com/none-standard-location/sitemap.xml</code> 

Read more about robots.txt here: http://www.seomoz.org/learn-seo/robotstxt.

[AshSEO2011]

Thanks for the answer and link John!

Regards,

Ash