Mobile site - allow robot traffic

[NicB1]

Hi,

If a user comes to our site from a mobile device, we redirect to our mobile site.  That is www.mysite/mypage redirects to m.mysite/mypage.  Right now we are blocking robots from crawling our m. site.  Previously there were concerns the m. site could rank for normal browser searches.  To make sure this isn't a problem we are planning on rel canonical our m. site pages and reference the www pages (mobile is just a different version of our www site).

From my understanding having a mobile version of a page is a ranking factor for mobile searches so allowing robots is a good thing.   Before doing so, I wanted to see if anyone had any other suggestions/feedback (looking for potential pitfalls, issues etc)

[john4math]

For Google's opinion, you'll want to watch http://www.youtube.com/watch?v=mY9h3G8Lv4k, and read http://googlewebmastercentral.blogspot.com/2009/11/help-google-index-your-mobile-site.html.

[dmac]

We don't block robots from our mobile site. Our site is designed to work well with iPad. We have found that our KWs perform slightly better when searched from a mobile device. It's a once little bonus for us. Duplicate content hasn't been a problem, I think the engines are smart enough to figure it out, even without a mobile sitemap.

[AdoptionHelp]

Rather than just blocking robots from crawling your mobile site, you should configure your robots.txt file to allow the mobile googlebot to crawl that subdomain specifically and disallow it on your regular domain. This way users searching on mobile devices will be shown your mobile site. In your current set up you risk sending them to your main site from the SERPs.

If you need help getting that set up there is a tool in Google webmasters called "Generate robots.txt" located in the crawler settings under Site Configuration.

[ChapterEight]

My preference is not to redirect to m.domain.com as there are potential issues with m.domain.com listings appearing in natural search. I would use the user agent detection, I think this gives the best user experience as well, rather than re-directing, what type of re-direct are you using anyway?

I would for best practice, still specify google mobile bot to crawl the mobile version of your website only ans restrict normal g bot form the mobile version.

Cheers,

Jamie.

[MWM3772]

Hi Jamie -- I noticed the mobile version of our website is appearing in the Bing search results (from a desktop) --- can you employ the same technique as Google and specify to the Bing/Yahoo! mobile bot to only crawl the mobile version of your site and restrict the normal bot from crawling mobile version?  Thanks for additional guidance.  I assume this would clear up this issue of our mobile site appearing in normal non-mobile search results.

• Matt