The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Robots.txt Download vs Cache

    Robots.txt Download vs Cache

    Technical SEO Issues
    2 2 128
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Rich_A
      Rich_A last edited by

      We made an update to the Robots.txt file this morning after the initial download of the robots.txt file.  I then submitted the page through Fetch as Google bot to get the changes in asap.

      The cache time stamp on the page now shows Sep 27, 2013 15:35:28 GMT.  I believe that would put the cache time stamp at about 6 hours ago.  However the Blocked URLs tab in Google WMT shows the robots.txt last downloaded at 14 hours ago - and therefore it's showing the old file.

      This leads me to believe for the Robots.txt the cache date and the download time are independent.  Is there anyway to get Google to recognize the new file other than waiting this out??

      1 Reply Last reply Reply Quote 0
      • FedeEinhorn
        FedeEinhorn last edited by

        No to my knowledge. You will have to wait. Anyway, Google could have already download the new robots but while the reports are showing the older file. Those reports always take a while until refreshing completely.

        1 Reply Last reply Reply Quote 1
        • 1 / 1
        • First post
          Last post
        • Website URL, Robots.txt and Google Search Console (www. vs non www.)
          RossKernez
          RossKernez
          0
          10
          121

        • Robots.txt on http vs. https
          BlueprintMarketing
          BlueprintMarketing
          0
          4
          7.0k

        • Easy Question: regarding no index meta tag vs robot.txt
          Everett
          Everett
          0
          4
          213

        • Robots.txt
          Dan-Lawrence
          Dan-Lawrence
          0
          5
          99

        • Robots.txt
          irvingw
          irvingw
          0
          4
          116

        • Robots.txt
          BailHotline
          BailHotline
          0
          5
          760

        • Robots.txt
          JordanGodbey
          JordanGodbey
          0
          6
          619

        • Robots.txt and robots meta
          TheEspresseo
          TheEspresseo
          0
          5
          1.1k

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy