The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Robots.txt blocking site or not?

    Robots.txt blocking site or not?

    Technical SEO Issues
    2 2 445
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • 540SEO
      540SEO last edited by

      Here is the robots.txt from a client site. Am I reading this right --
      that the robots.txt is saying to ignore the entire site, but the
      #'s are saying to ignore the robots.txt command?

      See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file

      To ban all spiders from the entire site uncomment the next two lines:

      User-Agent: *

      Disallow: /

      
      
      1 Reply Last reply Reply Quote 0
      • RyanKent
        RyanKent last edited by

        You are reading it correctly.

        Any text prefaced by a # character is ignored. The # symbol indicates a comment.

        More details are available at http://www.robotstxt.org/

        1 Reply Last reply Reply Quote 3
        • 1 / 1
        • First post
          Last post
        • Blocking subdomains with Robots.txt file
          PaulM01
          PaulM01
          0
          3
          641

        • Should you use robots.txt for pages within your site which do not have high quality content or are not contributing a great deal so when Google crawls your site the best performing content has a higher chance of being indexed?
          Jacksons_Fencing
          Jacksons_Fencing
          0
          5
          44

        • Google Indexing Development Site Despite Robots.txt Block
          DeanAndrews
          DeanAndrews
          0
          6
          990

        • Robots.txt & Mobile Site
          LynnPatchett
          LynnPatchett
          0
          3
          4.1k

        • Warnings for blocked by blocked by meta-robots/meta robots Nofollow...how to resolve?
          Cyrus-Shepard
          Cyrus-Shepard
          0
          3
          415

        • Blocking robots.txt
          de4e
          de4e
          0
          4
          432

        • Client accidently blocked entire site with robots.txt for a week
          KeriMorgret
          KeriMorgret
          0
          4
          1.4k

        • Search Engine blocked by robots.txt
          Goetzman
          Goetzman
          0
          7
          1.2k

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy