The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Suggested Screaming Frog configuration to mirror default Googlebot crawl?

    Suggested Screaming Frog configuration to mirror default Googlebot crawl?

    Intermediate & Advanced SEO
    1 1 68
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • gravymatt-se
      gravymatt-se last edited by

      Hi All,

      Does anyone have a suggested Screaming Frog  (SF) configuration to mirror default Googlebot crawl? I want to test my site and see if it will return 429 "Too Many Requests" to Google.

      I have set the User Agent as Googlebot (Smartphone). Is the default SF Menu > Configuration > Speed > Max Threads 5 and Max URLs 2.0 comparable to Googlebot?

      Context:
      I had tried NetPeak SEO Spider which did a nice job and had a cool feature that would pause a crawl if it got to many 429. Long Story short, B2B site  threw 429 Errors when there should have been no load on a holiday weekend at 1:00 AM.

      1 Reply Last reply Reply Quote 0
      • 1 / 1
      • First post
        Last post
      • Crawl Stats Decline After Site Launch (Pages Crawled Per Day, KB Downloaded Per Day)
        R0bin_L0rd
        R0bin_L0rd
        0
        4
        158

      • Site's pages has GA codes based on Tag Manager but in Screaming Frog, it is not recognized
        Martijn_Scheijbeler
        Martijn_Scheijbeler
        0
        2
        357

      • Screaming frog Advice
        Andy-Halliday
        Andy-Halliday
        0
        4
        566

      • Strange 404s in Screaming Frog
        screamingfrog
        screamingfrog
        0
        4
        148

      • Can't crawl website with Screaming frog... what is wrong?
        Singularitie
        Singularitie
        0
        3
        3.8k

      • Would you rate-control Googlebot? How much crawling is too much crawling?
        CraigBradford
        CraigBradford
        0
        4
        807

      • After Receiving a "Googlebot can't access your site" would this stop your site from being crawled?
        evolvingSEO
        evolvingSEO
        0
        4
        394

      • Googlebot crawling partial URLs
        Improvements
        Improvements
        0
        4
        1.2k

      Get started with Moz Pro!

      Unlock the power of advanced SEO tools and data-driven insights.

      Start my free trial
      Products
      • Moz Pro
      • Moz Local
      • Moz API
      • Moz Data
      • STAT
      • Product Updates
      Moz Solutions
      • SMB Solutions
      • Agency Solutions
      • Enterprise Solutions
      • Digital Marketers
      Free SEO Tools
      • Domain Authority Checker
      • Link Explorer
      • Keyword Explorer
      • Competitive Research
      • Brand Authority Checker
      • Local Citation Checker
      • MozBar Extension
      • MozCast
      Resources
      • Blog
      • SEO Learning Center
      • Help Hub
      • Beginner's Guide to SEO
      • How-to Guides
      • Moz Academy
      • API Docs
      About Moz
      • About
      • Team
      • Careers
      • Contact
      Why Moz
      • Case Studies
      • Testimonials
      Get Involved
      • Become an Affiliate
      • MozCon
      • Webinars
      • Practical Marketer Series
      • MozPod
      Connect with us

      Contact the Help team

      Join our newsletter
      Moz logo
      © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
      • Accessibility
      • Terms of Use
      • Privacy