The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Robots.txt

    Robots.txt

    Technical SEO Issues
    4 4 116
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • iskq
      iskq last edited by

      www.mywebsite.com**/details/**home-to-mome-4596

      www.mywebsite.com**/details/**home-moving-4599

      www.mywebsite.com**/details/**1-bedroom-apartment-4601

      www.mywebsite.com**/details/**4-bedroom-apartment-4612  We have so many pages like this, we do not want to Google crawl this pages

      So we added the following code to Robots.txt

      User-agent: Googlebot

      Disallow: /details/

      This code is correct?

      1 Reply Last reply Reply Quote 0
      • Chris.Menke
        Chris.Menke last edited by

        Looks good but you only want to exclude google from crawling those pages?  If you want to exclude other search engines, you could use:

        User-agent: *

        instead of

        User-agent: Googlebot

        KevinBudzynski 1 Reply Last reply Reply Quote 2
        • KevinBudzynski
          KevinBudzynski @Chris.Menke last edited by

          Looks correct. Great reccomendation by Chris. The * (wildcard) just means "all".

          1 Reply Last reply Reply Quote 0
          • irvingw
            irvingw last edited by

            and this disallows everything in the /details/ folder so if there were some exceptions to the rule (some pages or sub folders in that folder) you would need to add some allow directives, or make a more specific disallow(s)

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • Robots.txt
              MichaelC-15022
              MichaelC-15022
              0
              7
              1.0k

            • Robots.txt and Magento
              Felip3
              Felip3
              0
              6
              746

            • Robots.txt
              WesleySmits
              WesleySmits
              0
              9
              168

            • Robots.txt file
              Asher
              Asher
              0
              3
              261

            • Do I need robots.txt and meta robots?
              Cyrus-Shepard
              Cyrus-Shepard
              0
              7
              1.1k

            • Robots.txt question
              KeriMorgret
              KeriMorgret
              0
              7
              791

            • Robots.txt
              Ontarioseo
              Ontarioseo
              0
              5
              737

            • Robots.txt
              Entrusteddev
              Entrusteddev
              0
              3
              642

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy