The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Do I have a robots.txt problem?

    Do I have a robots.txt problem?

    Technical SEO Issues
    6 3 168
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • EcommerceSite
      EcommerceSite last edited by

      I have the little yellow exclamation point under my robots.txt fetch as you can see here- http://imgur.com/wuWdtvO

      This version shows no errors or warnings- http://imgur.com/uqbmbug

      Under the tester I can currently see the latest version. This site hasn't changed URLs recently, and we haven't made any changes to the robots.txt file for two years. This problem just started in the last month. Should I worry?

      1 Reply Last reply Reply Quote 0
      • Ray-pp
        Ray-pp last edited by

        Can you post your domain? Manually checking the robots.txt file would help.

        I've checked many of my GWT accounts and I am not showing a sudden robots.txt error. It could be a false error, but I would take anything with the robots.txt file seriously. You'll want to make sure that it is in fact accessible to all the crawlers desired.

        EcommerceSite 1 Reply Last reply Reply Quote 2
        • EcommerceSite
          EcommerceSite @Ray-pp last edited by

          I can't post the domain but I know it is accessible.

          When I go to the tester it shows the live robots.txt with no problems. I also can look at the server logs and see that it is being crawled, but being crawled less then Bing Crawls. Also the Bing Webmaster Tools is showing no problems.

          1 Reply Last reply Reply Quote 1
          • Ray-pp
            Ray-pp last edited by

            That would make me feel more confident of a false error being reported. Time to closely monitor the crawl logs, look at server stats, and keep an eye on GWT for a change in the reporting/indexing. I would also go into the GWT forums and post, see if anyone is reporting a similar error these past couple days.

            1 Reply Last reply Reply Quote 2
            • Martijn_Scheijbeler
              Martijn_Scheijbeler last edited by

              It could be that your server had a hard time when Google tried to view your robots.txt file that's why it wouldn't be able to fetch it. As long as this issue doesn't prevent Google anymore in the future it's not much to worry about.

              1 Reply Last reply Reply Quote 1
              • EcommerceSite
                EcommerceSite last edited by

                Today it has a green check mark, and absolutely no changes were made to the website since I asked this question.

                1 Reply Last reply Reply Quote 1
                • 1 / 1
                • First post
                  Last post
                • I have two robots.txt pages for www and non-www version. Will that be a problem?
                  jasongmcmahon
                  jasongmcmahon
                  0
                  6
                  206

                • Robots.txt
                  MarieHaynes
                  MarieHaynes
                  0
                  8
                  115

                • Meta Robots Noindex and Robots.txt File
                  Devanur-Rafi
                  Devanur-Rafi
                  0
                  2
                  125

                • Robots.txt
                  WesleySmits
                  WesleySmits
                  0
                  9
                  168

                • Robots.txt
                  irvingw
                  irvingw
                  0
                  4
                  116

                • Removing robots.txt on WordPress site problem
                  nextlevelweb
                  nextlevelweb
                  0
                  21
                  38.1k

                • Restricted by robots.txt does this cause problems?
                  Everett
                  Everett
                  0
                  3
                  487

                • Robots.txt
                  JordanGodbey
                  JordanGodbey
                  0
                  6
                  619

                Get started with Moz Pro!

                Unlock the power of advanced SEO tools and data-driven insights.

                Start my free trial
                Products
                • Moz Pro
                • Moz Local
                • Moz API
                • Moz Data
                • STAT
                • Product Updates
                Moz Solutions
                • SMB Solutions
                • Agency Solutions
                • Enterprise Solutions
                • Digital Marketers
                Free SEO Tools
                • Domain Authority Checker
                • Link Explorer
                • Keyword Explorer
                • Competitive Research
                • Brand Authority Checker
                • Local Citation Checker
                • MozBar Extension
                • MozCast
                Resources
                • Blog
                • SEO Learning Center
                • Help Hub
                • Beginner's Guide to SEO
                • How-to Guides
                • Moz Academy
                • API Docs
                About Moz
                • About
                • Team
                • Careers
                • Contact
                Why Moz
                • Case Studies
                • Testimonials
                Get Involved
                • Become an Affiliate
                • MozCon
                • Webinars
                • Practical Marketer Series
                • MozPod
                Connect with us

                Contact the Help team

                Join our newsletter
                Moz logo
                © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                • Accessibility
                • Terms of Use
                • Privacy