The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Why are these results being showed as blocked by robots.txt?

    Why are these results being showed as blocked by robots.txt?

    Intermediate & Advanced SEO
    9 4 203
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • nicole.healthline
      nicole.healthline last edited by

      If you perform this search, you'll see all m. results are blocked by robots.txt: http://goo.gl/PRrlI, but when I reviewed the robots.txt file: http://goo.gl/Hly28, I didn't see anything specifying to block crawlers from these pages.

      Any ideas why these are showing as blocked?

      1 Reply Last reply Reply Quote 0
      • Johnny4B
        Johnny4B last edited by

        Oh, sorry, on further investigation I see its just  your mobile site that are being blocked ...

        Johnny4B 1 Reply Last reply Reply Quote 0
        • Johnny4B
          Johnny4B @Johnny4B last edited by

          Which directory are you storing your mobile website files within ...

          1 Reply Last reply Reply Quote 0
          • Johnny4B
            Johnny4B last edited by

            So after a little it or research as I never ever came past this before as all the site we do are responsive, I found this

            http://support.google.com/webmasters/bin/answer.py?hl=en&answer=72462

            It seems Google wont index a website that they think is a mobile website within the main serp, and vice verse ...

            Hope that helps, cause it had me puzzled

            Regards

            John

            nicole.healthline 1 Reply Last reply Reply Quote 0
            • Johnny4B
              Johnny4B last edited by

              Another good article from the community

              http://www.seomoz.org/blog/mobile-serps-usability

              1 Reply Last reply Reply Quote 0
              • StreamlineMetrics
                StreamlineMetrics last edited by

                If you haven't already done so, I recommend testing your robots.txt file against one of your mobile pages (such as m.healthline.com/treatments) in Google Webmaster Tools. You can do this by logging into GWT, then click Health, then Blocked URLs.

                If you have already tested it in GWT, can you let us know what the results said?

                nicole.healthline 1 Reply Last reply Reply Quote 0
                • nicole.healthline
                  nicole.healthline @StreamlineMetrics last edited by

                  I can't submit because I haven't claimed m. in GWT

                  1 Reply Last reply Reply Quote 0
                  • nicole.healthline
                    nicole.healthline @Johnny4B last edited by

                    but, even when i search from my mobile device, I get the same results (that m. is blocked)

                    1 Reply Last reply Reply Quote 0
                    • eyepaq
                      eyepaq last edited by

                      Hi,

                      Your robots.txt file is very .. steroid healthy. It has his own universe 🙂

                      Are you 100% sure all of the entries are legit and clean ?

                      First thing I would do is to check Web M;aster Tools for the mobile subdomain. If you don't have it yet, that will be a good place to start - to verify the m subdomain.

                      Once in WeB Master Tools - you can debug this in no time.

                      Cheers.

                      1 Reply Last reply Reply Quote 1
                      • 1 / 1
                      • First post
                        Last post
                      • Search Results Pages Blocked in Robots.txt?
                        BeckyKey
                        BeckyKey
                        0
                        3
                        117

                      • Robots.txt Blocking - Best Practices
                        ReunionMarketing
                        ReunionMarketing
                        0
                        7
                        456

                      • Block subdomain directory in robots.txt
                        DirkC
                        DirkC
                        0
                        5
                        1.1k

                      • Block in robots.txt instead of using canonical?
                        RobertFisher
                        RobertFisher
                        0
                        9
                        1.6k

                      • What should I block with a robots.txt file?
                        Travis-W
                        Travis-W
                        1
                        3
                        298

                      • Files blocked in robot.txt and seo
                        john4math
                        john4math
                        0
                        4
                        344

                      • Block an entire subdomain with robots.txt?
                        kylesuss
                        kylesuss
                        1
                        16
                        102.1k

                      • Blocking Dynamic URLs with Robots.txt
                        TaitLarson
                        TaitLarson
                        1
                        4
                        5.1k

                      Get started with Moz Pro!

                      Unlock the power of advanced SEO tools and data-driven insights.

                      Start my free trial
                      Products
                      • Moz Pro
                      • Moz Local
                      • Moz API
                      • Moz Data
                      • STAT
                      • Product Updates
                      Moz Solutions
                      • SMB Solutions
                      • Agency Solutions
                      • Enterprise Solutions
                      • Digital Marketers
                      Free SEO Tools
                      • Domain Authority Checker
                      • Link Explorer
                      • Keyword Explorer
                      • Competitive Research
                      • Brand Authority Checker
                      • Local Citation Checker
                      • MozBar Extension
                      • MozCast
                      Resources
                      • Blog
                      • SEO Learning Center
                      • Help Hub
                      • Beginner's Guide to SEO
                      • How-to Guides
                      • Moz Academy
                      • API Docs
                      About Moz
                      • About
                      • Team
                      • Careers
                      • Contact
                      Why Moz
                      • Case Studies
                      • Testimonials
                      Get Involved
                      • Become an Affiliate
                      • MozCon
                      • Webinars
                      • Practical Marketer Series
                      • MozPod
                      Connect with us

                      Contact the Help team

                      Join our newsletter
                      Moz logo
                      © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                      • Accessibility
                      • Terms of Use
                      • Privacy