Robots.txt & Mobile Site

[CeeC-Blogger]

Background -  Our mobile site is on the same domain as our main site. We use a folder approach for our mobile site abc.com/m/home.html We are re-directing traffic to our mobile site vie device detection and re-direction exists for a handful of pages of our site ie most of our pages do not redirect the user to a mobile equivalent page.

Issue – Our mobile pages are being indexed in desktop Google searches

Input Required – How should we modify our robots.txt so that the desktop google index does not index our mobile pages/urls

User-agent: Googlebot-Mobile

Disallow: /m

User-agent: `YahooSeeker/M1A1-R2D2`

Disallow: /m

User-agent: `MSNBOT_Mobile`

Disallow: /m

Many thanks

[Saijo.George]

User-agent: Googlebot-Mobile

Disallow: /m

would't that statement prevent Googlebot-Mobile from crawling the mobile pages ? Why would you want to do that ?

[LynnPatchett]

Hi,

The fact that you do not have mobile versions for all pages of the site makes your case a bit of an odd one when it comes to best practices.  I would recommend if at all possible to address this and make all pages available in mobile format also.

If for whatever reason this is not possible then I think you are better off using a meta robots tag NOINDEX to get your mobile results out of the serps completely and a rel canonical to the relevant desktop page to make extra sure there are no issues with duplicate content.

If you can get all pages into a mobile version then you will want to implement proper redirects based on user string as you are doing, and also implement rel alternate/canonical tags between the desktop and mobile pages of the site (which should make sure the right version of the site shows up in the relevant serps). For more details on the whole setup check:

http://moz.com/blog/how-to-optimize-a-mobile-site and
http://moz.com/blog/mobile-seo-process