The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Set Robots.txt file to crawl my website at specific times

    Set Robots.txt file to crawl my website at specific times

    Intermediate & Advanced SEO
    2 2 50
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • CharleneKennedy12
      CharleneKennedy12 last edited by

      Our website provider has stated that they can only 'lift' their block on our website in order for it to be crawled as specific times.

      Is there any way to amend a robots.txt to ensure that it crawls our website at a specific time of day/night in order to coincide with the block being lifted?

      Many Thanks,

      Charlene

      1 Reply Last reply Reply Quote 0
      • Tenlo
        Tenlo last edited by

        Charlene,

        Your question does not make any sense, what block are they talking about? If your website provider is not allowing Google to crawl your site you should go look at getting a new provider straight away.

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        • Robots.txt was set to disallow for 14 days
          jc4254
          jc4254
          0
          3
          58

        • If my website do not have a robot.txt file, does it hurt my website ranking?
          Alick300
          Alick300
          0
          2
          242

        • How to make Google index your site? (Blocked with robots.txt for a long time)
          SanjidaKazi
          SanjidaKazi
          0
          3
          128

        • Issue with Robots.txt file blocking meta description
          ThompsonPaul
          ThompsonPaul
          0
          9
          3.8k

        • Is our robots.txt file correct?
          Igal_Zeifman
          Igal_Zeifman
          0
          5
          175

        • Files blocked in robot.txt and seo
          john4math
          john4math
          0
          4
          344

        • 202 error page set in robots.txt versus using crawl-able 404 error
          EricaMcGillivray
          EricaMcGillivray
          0
          3
          870

        • Negative impact on crawling after upload robots.txt file on HTTPS pages
          ShaMenz
          ShaMenz
          0
          2
          892

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy