The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Robots.txt | any SEO advantage to having one vs not having one?

    Robots.txt | any SEO advantage to having one vs not having one?

    Technical SEO Issues
    4 4 119
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • GregB123
      GregB123 last edited by

      Neither of my sites has a robots.txt file. I guess I have never been bothered by any particular bot enough to exclude it.

      Is there any SEO advantage to having one anyways?

      1 Reply Last reply Reply Quote 0
      • OlegKorneitchouk
        OlegKorneitchouk last edited by

        Just good practice. One SEO advantage would be to include a reference to your sitemap within the robots.txt file.

        Aside from that, if you want all of your pages crawled and don't have a sitemap (although you should), no need for a robots.txt file.

        1 Reply Last reply Reply Quote 0
        • DennisSeymour
          DennisSeymour last edited by

          There are actually a couple good reasons but in short, it's "best practice" so it won't hurt by adding it in. It wont take more than a couple minutes.

          1 Reply Last reply Reply Quote 0
          • JaneCopland
            JaneCopland last edited by

            It's good practice, especially if you are operating a CMS that can create accessible URLs that cause duplicate content problems, create "junk" pages, etc. For example: http://www.asos.com/robots.txt

            Google dislikes search results pages being indexed, so you can block those off, e.g. http://moz.com/robots.txt

            You can disallow the archive.org bot if you don't want old versions of your site appearing in its search engine, and as others have said you can point to your xml sitemap.

            It's not a bad resource to have at your disposal for site hygiene / maintenance reasons, but it's not an absolute necessity either.

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • One robots.txt file for multiple sites?
              RyanPurkey
              RyanPurkey
              0
              4
              1.1k

            • Easy Question: regarding no index meta tag vs robot.txt
              Everett
              Everett
              0
              4
              213

            • Is having no robots.txt file the same as having one and allowing all agents?
              ITRIX
              ITRIX
              0
              2
              697

            • Do I need robots.txt and meta robots?
              Cyrus-Shepard
              Cyrus-Shepard
              0
              7
              1.1k

            • Internal search : rel=canonical vs noindex vs robots.txt
              Dr-Pete
              Dr-Pete
              0
              9
              4.5k

            • SEO WP Plugin Battle: Yoast Wordpress SEO vs All in One SEO?
              vijayvasu
              vijayvasu
              0
              3
              735

            • URL Structure "-" vs "/"? Are there any advantages to one over the other?
              RyanKent
              RyanKent
              0
              5
              920

            • Robots.txt and robots meta
              TheEspresseo
              TheEspresseo
              0
              5
              1.1k

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy