The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Pro
    4. Can't work out robots.txt issue.

    Can't work out robots.txt issue.

    Moz Pro
    3 3 137
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • brenmcc1
      brenmcc1 last edited by

      Hi

      I'm getting crawl errors that MOZ isn't able to access my robots.txt file but it seems completely fine to me? Any chance anyone can help me understand what might be the issue?

      www.equip4gyms.co

      1 Reply Last reply Reply Quote 0
      • Roman-Delcarmen
        Roman-Delcarmen last edited by

        Ok, I made a quick test of your robot.txt file and looks fine, making HTTP status code test and shows me 200 code which is ok. Also, you need to make sure that your robot.txt file is accessible for the Moz crawler

        • Remember to put your file is in the top-level directory of your web server.
        • Also, check if your hosting provider is not blocking third-party crawlers (server level)

        Here you can test the status code of your robot.txt file** https://httpstatus.io/ **If you're still having trouble this is the email of Moz help@moz.com

        Also, you should check

        • https://moz.com/learn/seo/robotstxt
        • https://support.google.com/webmasters/answer/6062608?hl=en

        Best of luck!

        1 Reply Last reply Reply Quote 1
        • Martijn_Scheijbeler
          Martijn_Scheijbeler last edited by

          I would follow the advise from Roman, I tried to access the file via a few ways as well and it doesn't seem to be an issue to load it at all.

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Only crawling one page...still don't get why.
            Abe_Schmidt
            Abe_Schmidt
            0
            3
            190

          • Duplicate page reported in Wordpress site, but I can't find it in All Pages list
            BlueprintMarketing
            BlueprintMarketing
            0
            6
            360

          • I'm trying to get 'tigi bed head' up most of all...
            smoki666
            smoki666
            0
            7
            633

          • Why can't I add my facebook page to SEOMOZ? Also having other facebook issues.
            KeriMorgret
            KeriMorgret
            1
            11
            2.3k

          • Why does SEOMoz crawler ignore robots.txt?
            Keszi
            Keszi
            0
            4
            2.0k

          • I don't get what a dynamic URL is?
            evolvingSEO
            evolvingSEO
            0
            4
            623

          • Can't add my domain климатмонтаж.рф to Campaign Manager
            KeriMorgret
            KeriMorgret
            0
            2
            1.1k

          • Ranking reports don't seem to match reality
            krissy-cca
            krissy-cca
            0
            3
            1.0k

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy