The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Is there any value in having a blank robots.txt file?

    Is there any value in having a blank robots.txt file?

    Technical SEO Issues
    6 4 3.1k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • NicDale
      NicDale last edited by

      I've read an audit where the writer recommended creating and uploading a blank robots.txt file, there was no current file in place.  Is there any merit in having a blank robots.txt file?

      What is the minimum you would include in a basic robots.txt file?

      1 Reply Last reply Reply Quote 0
      • Modi
        Modi last edited by

        Hi, This is what i got

        " Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called_The Robots Exclusion Protocol_. It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

        User-agent: *
        Disallow: /
        
        

        The "<tt>User-agent: *</tt>" means this section applies to all robots. The "<tt>Disallow: /</tt>" tells the robot that it should not visit any pages on the site."

        More and more people use robots,txt to disallow access to some administration or private folders of the site . If you dont want to hide anything then may be you can leave it blank

        1 Reply Last reply Reply Quote 0
        • zeepartner
          zeepartner last edited by

          No use in having a blank robots.txt. Minimum requirement if you want to have your site crawled is this:

          User-agent: *
          Allow: /
          

          Note that Gagans example above will block the entire site.

          Modi 1 Reply Last reply Reply Quote 1
          • Modi
            Modi @zeepartner last edited by

            i mentioned same only, The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site."

            n has added - More and more people use robots,txt to disallow access to some administration or private folders of the site

            1 Reply Last reply Reply Quote 0
            • NicDale
              NicDale last edited by

              Thanks for both of your replies.  As per my question it was around whether there is any value having a blank robots.txt file.  Philipp's answer was right on the money.

              1 Reply Last reply Reply Quote 0
              • LiamVictor
                LiamVictor last edited by

                I know this is four years old, but there's value in having a blank robots.txt as some tools (including the latest version of the Moz crawler) will baulk at sites without a robots.txt file.

                1 Reply Last reply Reply Quote 0
                • 1 / 1
                • First post
                  Last post
                • Multiple robots.txt files on server
                  peakdistrictseo
                  peakdistrictseo
                  0
                  8
                  2.4k

                • Does this robots.txt file look right?
                  ThompsonPaul
                  ThompsonPaul
                  0
                  8
                  188

                • Question about construction of our sitemap URL in robots.txt file
                  danatanseo
                  danatanseo
                  0
                  16
                  1.6k

                • Do i have my robots.txt file set up properly
                  ClaireH-184886
                  ClaireH-184886
                  1
                  4
                  319

                • Is my robots.txt file working?
                  Resultify
                  Resultify
                  0
                  5
                  1.1k

                • Use of Robots.txt file on a job site
                  jennita
                  jennita
                  0
                  5
                  850

                • Robots.txt and robots meta
                  TheEspresseo
                  TheEspresseo
                  0
                  5
                  1.1k

                • Robots.txt File Redirects to Home Page
                  NickPateman81
                  NickPateman81
                  0
                  6
                  4.1k

                Get started with Moz Pro!

                Unlock the power of advanced SEO tools and data-driven insights.

                Start my free trial
                Products
                • Moz Pro
                • Moz Local
                • Moz API
                • Moz Data
                • STAT
                • Product Updates
                Moz Solutions
                • SMB Solutions
                • Agency Solutions
                • Enterprise Solutions
                • Digital Marketers
                Free SEO Tools
                • Domain Authority Checker
                • Link Explorer
                • Keyword Explorer
                • Competitive Research
                • Brand Authority Checker
                • Local Citation Checker
                • MozBar Extension
                • MozCast
                Resources
                • Blog
                • SEO Learning Center
                • Help Hub
                • Beginner's Guide to SEO
                • How-to Guides
                • Moz Academy
                • API Docs
                About Moz
                • About
                • Team
                • Careers
                • Contact
                Why Moz
                • Case Studies
                • Testimonials
                Get Involved
                • Become an Affiliate
                • MozCon
                • Webinars
                • Practical Marketer Series
                • MozPod
                Connect with us

                Contact the Help team

                Join our newsletter
                Moz logo
                © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                • Accessibility
                • Terms of Use
                • Privacy