The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Pro
    4. Can't work out robots.txt issue.

    Can't work out robots.txt issue.

    Moz Pro
    3 3 137
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • brenmcc1
      brenmcc1 last edited by

      Hi

      I'm getting crawl errors that MOZ isn't able to access my robots.txt file but it seems completely fine to me? Any chance anyone can help me understand what might be the issue?

      www.equip4gyms.co

      1 Reply Last reply Reply Quote 0
      • Roman-Delcarmen
        Roman-Delcarmen last edited by

        Ok, I made a quick test of your robot.txt file and looks fine, making HTTP status code test and shows me 200 code which is ok. Also, you need to make sure that your robot.txt file is accessible for the Moz crawler

        • Remember to put your file is in the top-level directory of your web server.
        • Also, check if your hosting provider is not blocking third-party crawlers (server level)

        Here you can test the status code of your robot.txt file** https://httpstatus.io/ **If you're still having trouble this is the email of Moz help@moz.com

        Also, you should check

        • https://moz.com/learn/seo/robotstxt
        • https://support.google.com/webmasters/answer/6062608?hl=en

        Best of luck!

        1 Reply Last reply Reply Quote 1
        • Martijn_Scheijbeler
          Martijn_Scheijbeler last edited by

          I would follow the advise from Roman, I tried to access the file via a few ways as well and it doesn't seem to be an issue to load it at all.

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Robots.txt
            OlegKorneitchouk
            OlegKorneitchouk
            0
            2
            129

          • Why my site page don't appeare in the ranking ?
            RepLoc_Tim
            RepLoc_Tim
            0
            3
            133

          • Duplicate page reported in Wordpress site, but I can't find it in All Pages list
            BlueprintMarketing
            BlueprintMarketing
            0
            6
            360

          • Links don't add up
            landmark1
            landmark1
            0
            7
            582

          • Why does SEOMoz crawler ignore robots.txt?
            Keszi
            Keszi
            0
            4
            2.0k

          • Fetch googlebot for sites you don't own?
            loopyal
            loopyal
            0
            4
            2.8k

          • Google UK rankings don't seem to be accurate.
            EdwardUpton61
            EdwardUpton61
            0
            3
            636

          • Ranking reports don't seem to match reality
            krissy-cca
            krissy-cca
            0
            3
            1.0k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy