The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. How many times robots.txt gets visited by crawlers, especially Google?

    How many times robots.txt gets visited by crawlers, especially Google?

    Technical SEO Issues
    2 2 237
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • linklater
      linklater last edited by

      Hi,

      Do you know if there's any way to track how often robots.txt file has been crawled?

      I know we can check when is the latest downloaded from webmaster tool, but I actually want to know if they download every time crawlers visit any page on the site (e.g. hundreds of thousands of times every day), or less.

      thanks...

      1 Reply Last reply Reply Quote 0
      • RyanKent
        RyanKent last edited by

        Your web server logs keep track of every file that is serves and to whom it provides the data. The raw log files are difficult to read unless you are used to viewing that type of code. The process varies based on your server type. Ask your web host or developer for guidance.

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        • Is there a limit to how many URLs you can put in a robots.txt file?
          CraigBradford
          CraigBradford
          0
          10
          1.1k

        • No descripton on Google/Yahoo/Bing, updated robots.txt - what is the turnaround time or next step for visible results?
          adamhdrb
          adamhdrb
          0
          13
          618

        • Google couldn't follow your URL because it redirected too many times.
          Everett
          Everett
          0
          7
          1.8k

        • Google index dymamic webpages after block in robots.txt...
          Dr-Pete
          Dr-Pete
          0
          6
          247

        • Google insists robots.txt is blocking... but it isn't.
          Christy-Correll
          Christy-Correll
          0
          4
          523

        • I accidentally blocked Google with Robots.txt. What next?
          SebastianCowie
          SebastianCowie
          0
          7
          2.1k

        • Google (GWT) says my homepage and posts are blocked by Robots.txt
          stubby
          stubby
          0
          8
          930

        • I have found this on a site that i have seen many times where can i get one from
          ClaireH-184886
          ClaireH-184886
          0
          5
          595

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy