The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Is Googlebot ignoring directives? Or is it Me?

    Is Googlebot ignoring directives? Or is it Me?

    Technical SEO Issues
    3 2 456
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • loopyal
      loopyal last edited by

      I saw an answer to a question in this forum a few days ago, that said it was a bad idea to use robots.txt to tell googlebot to go away.

      That SEO said it was much better to use the META tag to say noindex,nofollow.

      So I removed the robots directive and added the META tag

      <meta robots='noindex,nofollow'>
      

      Today, I see google showing my send to a friend page where I expected the real page to be.

      Does it mean Google is stupid?

      Does it mean google ignores the Robots META tag?

      Does it mean short pages have more value than long pages?

      Does it mean if I convert my whole site to snippets, I'll get more traffic?

      Does it mean garbage trumps content?

      I have more questions, but this is more than enough.

      1 Reply Last reply Reply Quote 0
      • RyanKent
        RyanKent last edited by

        Hi Allan,

        It is a best practice to use meta tags to indicate your indexing preference to search engines.

        Normally the recommended implementation would be "noindex, follow" but without examining your site it is impossible to know for sure.

        Google honors meta tags but there are a number of issues which could be the source of your issue. For example, if you did not use valid syntax the tag may not be honored. If you are blocking the page in robots.txt, then search engines cannot read the tag.

        As for the last three questions, the simple answer is quality content is best.

        If you can share the URL of the page involved, we can offer a specific response to the implementation of the meta tag.

        loopyal 1 Reply Last reply Reply Quote 3
        • loopyal
          loopyal @RyanKent last edited by

          Thank you Ryan.

          They completely ignored the meta tags., completely messing up our serps. So I put it back in robots. I wont trust google again to do the right thing.

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Googlebots and cache
            DirkC
            DirkC
            0
            2
            57

          • How can I make it so that robots.txt is not ignored due to a URL re-direct?
            rodelmo4
            rodelmo4
            0
            4
            56

          • Googlebot cannot access your site
            Rank-and-Grow
            Rank-and-Grow
            0
            3
            56

          • Are my canonical re directs working?
            Andy.Drinkwater
            Andy.Drinkwater
            0
            8
            254

          • Googlebot cannot access your site
            RyanKent
            RyanKent
            0
            5
            447

          • Rel=Canonical being ignored?
            Cyrus-Shepard
            Cyrus-Shepard
            0
            2
            509

          • Differences between Lynx Viewer, Fetch as Googlebot and SEOMoz Googlebot Rendering
            AlanBleiweiss
            AlanBleiweiss
            0
            2
            1.5k

          • Re-direct issues
            Doc_Sheldon
            Doc_Sheldon
            0
            4
            940

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy