The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical Support
    4. "Our crawler was not able to access the robots.txt file on your site."

    "Our crawler was not able to access the robots.txt file on your site."

    Technical Support
    2 2 151
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • K3Syspro
      K3Syspro last edited by

      Hi Mozzers!

      I've received an error message saying the site can't be crawled because Moz is unable to access the robots.txt. I've spoken to the webmaster and he can't understand why the robot.txt can't be accessed as this seems to be fine:

      https://k3syspro.com/robots.txt

      and Google isn't flagging anything up to us.

      Does anyone know why this may be?

      Thanks,

      Matthew

      1 Reply Last reply Reply Quote 0
      • wholewhaler
        wholewhaler last edited by

        Hey K3,

        Looks like you have bigger fish to fry, your host domain is throwing a 503 server error. No server = no content = no robots.txt  will be able to be found until that is resolved. Error still happening as of this post: http://downforeveryoneorjustme.com/https://k3syspro.com

        1 Reply Last reply Reply Quote 2
        • 1 / 1
        • First post
          Last post
        • Crawl error robots.txt
          Mandiram
          Mandiram
          0
          10
          688

        • Campaign Dashboard Error: "Our Connection to Your Google Account Has Been Lost"
          ViviCa1
          ViviCa1
          0
          3
          70

        • Keeping getting the same results for "link opportunities"
          Sean_Peerenboom
          Sean_Peerenboom
          0
          2
          72

        • Is there a way to alphabetize the "Manage your Campaigns" screen?
          BKraff
          BKraff
          0
          3
          61

        • Duplicate Content Report: Duplicate URLs being crawled with "++" at the end
          ChiarynMiranda
          ChiarynMiranda
          0
          6
          253

        • We are having difficulties connecting to your "Facebook" account.
          vmialik
          vmialik
          0
          2
          99

        • The websites I add to my Moz Analytics account do not appear in the old "Pro" account. What is going on here?
          KeriMorgret
          KeriMorgret
          1
          2
          82

        • "We are collecting your traffic data now!" message displayed for many days
          ChiarynMiranda
          ChiarynMiranda
          0
          10
          186

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy