The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Is having no robots.txt file the same as having one and allowing all agents?

    Is having no robots.txt file the same as having one and allowing all agents?

    Technical SEO Issues
    2 2 697
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • pugh
      pugh last edited by

      The site I am working on currently has no robots.txt file. However, I have just uploaded a sitemap and would like to point the robots.txt file to it.

      Once I upload the robots.txt file, if I allow access to all agents, is this the same as when the site had no robots.txt file at all; do I need to specify crawler access on can the robots.txt file just contain the link to the sitemap?

      1 Reply Last reply Reply Quote 0
      • ITRIX
        ITRIX last edited by

        According to me a sitemap is more important than robots.txt as it help a search engine bot in effectively crawling a website. Robots.txt is generally used to request (allow: or disallow:)a crawler not to crawl and index certain section of your website containing sensitive data. This is totally upto the crawler to respect the request by not crawling and indexing that sensitive part. However, it is a general practice among webmasters world wide to have a robots.txt file for each of their sites.  A common robots.txt with permission to access the entire website should look like this:

        User-agent: *
        Disallow:

        Sitemap: http://www.yoursite.com/sitemap.xml

        So if you want some section (folders, directories) of your site not to be crawled by a bot then you can use a robots.txt.

        Yes logically its the same like having a robots.txt file granting all the access and not having one completely. Its just a difference between like something having 'by default". Having a robots.txt file doesn't guarantee a rank boost in the SERP. Hope it helps. For more understanding please refer these resources:

        http://www.seomoz.org/learn-seo/robotstxt

        http://www.robotstxt.org/robotstxt.html

        Cheers 🙂

        1 Reply Last reply Reply Quote 2
        • 1 / 1
        • First post
          Last post
        • Does putting Disallow: / at the end of a robots.txt file override the Allow: /xxxx that come before it?
          Alick300
          Alick300
          0
          4
          249

        • Multiple robots.txt files on server
          peakdistrictseo
          peakdistrictseo
          0
          8
          2.4k

        • Meta Robots Noindex and Robots.txt File
          Devanur-Rafi
          Devanur-Rafi
          0
          2
          125

        • Is my robots.txt file working?
          Resultify
          Resultify
          0
          5
          1.1k

        • How to allow one directory in robots.txt
          EricaMcGillivray
          EricaMcGillivray
          0
          2
          444

        • Use of Robots.txt file on a job site
          jennita
          jennita
          0
          5
          850

        • Robots.txt and robots meta
          TheEspresseo
          TheEspresseo
          0
          5
          1.1k

        • Robots.txt File Redirects to Home Page
          NickPateman81
          NickPateman81
          0
          6
          4.1k

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy