The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical Support
    4. "Our crawler was not able to access the robots.txt file on your site."

    "Our crawler was not able to access the robots.txt file on your site."

    Technical Support
    2 2 151
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • K3Syspro
      K3Syspro last edited by

      Hi Mozzers!

      I've received an error message saying the site can't be crawled because Moz is unable to access the robots.txt. I've spoken to the webmaster and he can't understand why the robot.txt can't be accessed as this seems to be fine:

      https://k3syspro.com/robots.txt

      and Google isn't flagging anything up to us.

      Does anyone know why this may be?

      Thanks,

      Matthew

      1 Reply Last reply Reply Quote 0
      • wholewhaler
        wholewhaler last edited by

        Hey K3,

        Looks like you have bigger fish to fry, your host domain is throwing a 503 server error. No server = no content = no robots.txt  will be able to be found until that is resolved. Error still happening as of this post: http://downforeveryoneorjustme.com/https://k3syspro.com

        1 Reply Last reply Reply Quote 2
        • 1 / 1
        • First post
          Last post
        • Crawl error robots.txt
          Mandiram
          Mandiram
          0
          10
          688

        • Site down: "high CPU usage is due to the large traffic generated from Moz
          jessential
          jessential
          0
          6
          181

        • Campaign Dashboard Error: "Our Connection to Your Google Account Has Been Lost"
          ViviCa1
          ViviCa1
          0
          3
          70

        • Keyword rankings "soon"?
          moz_support
          moz_support
          0
          2
          113

        • I plan to have multiple sites/compaigns. My Question: Is there a way to set up a limited account that one client would be able to view just their campaign?
          Ray-pp
          Ray-pp
          0
          2
          258

        • Keeping getting the same results for "link opportunities"
          Sean_Peerenboom
          Sean_Peerenboom
          0
          2
          72

        • Is there a way to alphabetize the "Manage your Campaigns" screen?
          BKraff
          BKraff
          0
          3
          61

        • Can't add "Custom links" in my Moz Profile
          Websnare-SEO
          Websnare-SEO
          0
          6
          769

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy