The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Search Engine Trends
    4. Google Unable to Access Robots.txt

    Google Unable to Access Robots.txt

    Search Engine Trends
    4 3 205
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • rhoadesjohn
      rhoadesjohn last edited by

      We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.

      The site has been up and active for well over a year now.  What are my next steps?

      I have included a screenshot of the top half of the file.  See anything wrong?

      D3H5tgE.png

      1 Reply Last reply Reply Quote 0
      • zeepartner
        zeepartner last edited by

        I think you should allow general access before beginning to disallow various sections below:

        Allow: /

        Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.

        1 Reply Last reply Reply Quote 0
        • rhoadesjohn
          rhoadesjohn last edited by

          Thanks Phillip.  I think we're on the way to discovering the issue.

          I certainly appreciate the feedback!

          1 Reply Last reply Reply Quote 0
          • Modi
            Modi last edited by

            Would you mind sharing the solution - if not related to domain intricacies

            Curious to know - only for knowledge purpose

            Many thanks

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • Need only tens of pages to be indexed out of hundreds: Robots.txt is Okay for Google to proceed with?
              GastonRiera
              GastonRiera
              0
              2
              24

            • Indexed, though blocked by robots.txt: Need to bother?
              GastonRiera
              GastonRiera
              1
              2
              121

            • Meta robots at every page rather than using robots.txt for blocking crawlers? How they'll get indexed if we block crawlers?
              ThompsonPaul
              ThompsonPaul
              0
              3
              253

            • What is the appropriate Robot.txt to unblock if Google cannot get all the resources from my homepage?
              OlegKorneitchouk
              OlegKorneitchouk
              0
              2
              5.7k

            • Google cant read my robots.txt from past 10 days
              toxicpls
              toxicpls
              0
              5
              152

            • Why are Google Webmaster Tools' Google rankings different to actual Google rankings?
              BoomDialogue69
              BoomDialogue69
              0
              3
              73

            • Webpage is ranking on google.ie / google.co.uk but not google.com?
              Dr-Pete
              Dr-Pete
              0
              7
              1.7k

            • Is it hurting my seo ranking if robots.txt is forbidden?
              Sebes
              Sebes
              0
              6
              1.3k

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy