Mobile site ranking in desktop searches

[craigsmith333]

A robots.txt file is properly implemented on the mobile site to block Googlebot from crawling, yet, when running a site: query for the site in google, it still returns over 104,000 pages from the mobile site in the index.

Why could this be happening?

[KeriMorgret]

How long has the robots.txt file been in place? If you added it just a few days ago, it takes time for things to get out of the index. Other thoughts are looking at the source code of the pages on the mobile site -- is there a robots parameter on the page that is overriding robots.txt?

Have you gone into GWT and used the robots verification tool there to make sure nothing was amiss?

If you can give us your URL, the community can take a look and see if they can figure anything out.

[craigsmith333]

Thanks for responding Keri!

I can confirm that there are no parameters overriding the robots.txt within the code. I also can confirm that there is nothing amiss with the verification of the robots.txt.

Also, unfortunately, due to the nature of the client, we are unable to put the URL out there.

If you have any other thoughts, please do share. Sorry I couldn't be of more help.

Thank you.

[KeriMorgret]

If you haven't asked a question in private Q&A yet this month, you can ask there and share your URL, as that is private and available to only SEOmoz staff and associates.

Have you gone into GWT and requested removal of the site? If you verify the site, and then have the site/directory blocked via robots.txt, you can use the removal tool to request removal rather than waiting for things to drop out of the index.

[ClassifiedsKing]

I have thse recomendations:

1. Specify your Doctype to help Google see what your content type is. For mobile Google supports XHTML Basic, XHTML Mobile Profile, XHTML, WML, cHTML (iMode), and EZweb (KDDI). We run HTML 5, CSS 3 and jQuery successfully but these are not specified by google, so i would recommend to be safe and go with what Google supports.

2. Add a specialized XML Sitemap for Mobile. It is basically the same as the usual Sitemap apart from adding these to elements: xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0 and mobile:mobileRead more here: http://www.google.com/support/webmasters/bin/answer.py?answer=34648</mobile:mobile>

P.S. Name it mobilesitemap.xml.

3. Use WURFL to redirect PC users finding mobi content in the www index back to the PC version.

That should help.