The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Is there a reason to set a crawl-delay in the robots.txt?

    Is there a reason to set a crawl-delay in the robots.txt?

    Technical SEO Issues
    2 2 737
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • MichaelWeisbaum
      MichaelWeisbaum last edited by

      I've recently encountered a site that has set a crawl-delay command set in their robots.txt file.  I've never seen a need for this to be set since you can set that in Google Webmaster Tools for Googlebot.  They have this command set for all crawlers, which seems odd to me.  What are some reasons that someone would want to set it like that?  I can't find any good information on it when researching.

      1 Reply Last reply Reply Quote 0
      • NakulGoyal
        NakulGoyal last edited by

        Google does not support the crawl delay command directly, but you can lower your crawl priority inside Google Webmaster Central.

        So you are right using it the way you are. If you have it in the robots.txt, it does not really do anything and it will show in the webmaster console as well that GWT does not support it.

        1 Reply Last reply Reply Quote -1
        • 1 / 1
        • First post
          Last post
        • Crawl solutions for landing pages that don't contain a robots.txt file?
          Nomader
          Nomader
          1
          10
          353

        • Should you use robots.txt for pages within your site which do not have high quality content or are not contributing a great deal so when Google crawls your site the best performing content has a higher chance of being indexed?
          Jacksons_Fencing
          Jacksons_Fencing
          0
          5
          44

        • Robots.txt
          MichaelC-15022
          MichaelC-15022
          0
          7
          1.0k

        • Meta Robots Noindex and Robots.txt File
          Devanur-Rafi
          Devanur-Rafi
          0
          2
          125

        • Do i have my robots.txt file set up properly
          ClaireH-184886
          ClaireH-184886
          1
          4
          319

        • Robots.txt
          BailHotline
          BailHotline
          0
          5
          760

        • Should I set up a disallow in the robots.txt for catalog search results?
          AlanBleiweiss
          AlanBleiweiss
          0
          5
          1.7k

        • Robots.txt and robots meta
          TheEspresseo
          TheEspresseo
          0
          5
          1.1k

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy