The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Tools
    4. Our crawler was not able to access the robots.txt file on your site.

    Our crawler was not able to access the robots.txt file on your site.

    Moz Tools
    3 3 80
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Danny_Gallagher
      Danny_Gallagher last edited by

      Good morning,

      Yesterday, Moz gave me an error that is wasn't able to find our robots.txt file. However, this is a new occurrence, we've used Moz and its crawling ability many times prior; not sure why the error is happening now. I validated that the redirects and our robots page are operational and nothing is disallowing Roger in our robots.txt.

      Any advice or guidance would be much appreciated.

      https://www.agrisupply.com/robots.txt

      Thank you for your time.

      -Danny

      1 Reply Last reply Reply Quote 0
      • Jason-Rogers
        Jason-Rogers last edited by

        Try changing the user agent to googlebot and see if it works

        1 Reply Last reply Reply Quote 0
        • Optimal_Strategies
          Optimal_Strategies last edited by

          Hi,

          Aren't you just allowing Googlebot to scan /images folder?

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Robots.txt file issues on Shopify server
            Expansyon
            Expansyon
            0
            2
            44

          • Robots.txt blocking Moz
            ChristinaRadisic
            ChristinaRadisic
            0
            6
            291

          • Htaccess and robots.txt and 902 error
            SEOguy1
            SEOguy1
            0
            6
            1.1k

          • Moz campaign works around my robots.txt settings
            VinceWicks
            VinceWicks
            0
            4
            149

          • SEOmoz crawler not crawling my site
            IMM
            IMM
            0
            5
            787

          • Does Rogerbot respect the robots.txt file for wildcards?
            AC_Pro
            AC_Pro
            0
            4
            574

          • To block with robots.txt or canonicalize?
            STPseo
            STPseo
            0
            2
            522

          • SEOMoz site crawlers created an issue for our servers
            G-Force
            G-Force
            0
            4
            992

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy