The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Moz Tools
    4. Our crawler was not able to access the robots.txt file on your site.

    Our crawler was not able to access the robots.txt file on your site.

    Moz Tools
    3 3 80
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Danny_Gallagher
      Danny_Gallagher last edited by

      Good morning,

      Yesterday, Moz gave me an error that is wasn't able to find our robots.txt file. However, this is a new occurrence, we've used Moz and its crawling ability many times prior; not sure why the error is happening now. I validated that the redirects and our robots page are operational and nothing is disallowing Roger in our robots.txt.

      Any advice or guidance would be much appreciated.

      https://www.agrisupply.com/robots.txt

      Thank you for your time.

      -Danny

      1 Reply Last reply Reply Quote 0
      • Jason-Rogers
        Jason-Rogers last edited by

        Try changing the user agent to googlebot and see if it works

        1 Reply Last reply Reply Quote 0
        • Optimal_Strategies
          Optimal_Strategies last edited by

          Hi,

          Aren't you just allowing Googlebot to scan /images folder?

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Robots.txt file issues on Shopify server
            Expansyon
            Expansyon
            0
            2
            44

          • Robots.txt blocking Moz
            ChristinaRadisic
            ChristinaRadisic
            0
            6
            291

          • SEOmoz crawler not crawling my site
            IMM
            IMM
            0
            5
            787

          • Why do crawlers still track meta keywords if it is not needed in my site?
            jernest002
            jernest002
            0
            6
            395

          • I am trying to find inbound links for one of my site urls. My question is does SEOMoz able to track all internal links as the Open Site Explorer shows 0 internal links?
            Jeepster
            Jeepster
            0
            4
            492

          • HTTPS site, server root Robots.txt and Crawl errors
            KeriMorgret
            KeriMorgret
            0
            3
            390

          • Seomoz bar: No Follow and Robots.txt
            squareplug
            squareplug
            0
            7
            1.3k

          • Why is Roger crawling pages that are disallowed in my robots.txt file?
            MeltButterySpread
            MeltButterySpread
            0
            5
            928

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy