The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Bingbot appears to be crawling a large site extremely frequently?

    Bingbot appears to be crawling a large site extremely frequently?

    Technical SEO Issues
    3 2 557
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • danatanseo
      danatanseo last edited by

      Hi All! What constitutes a normal crawl rate for daily bingbot server requests for large sites? Are any of you noticing spikes in Bingbot crawl activity?

      I did find a "mildly" useful thread at Black Hat World containing this quote: "The reason BingBot seems to be terrorizing your site is because of your site's architecture; it has to be misaligned. If you are like most people, you paid no attention to setting up your website to avoid this glitch. In the article referenced by Oxonbeef, the author's issue was that he was engaging in dynamic linking, which pretty much put the BingBot in a constant loop.

      You may have the same type or similar issue particularly if you set up a WP blog without setting the parameters for noindex from the get go."

      However, my gut instinct says this isn't it and that it's more likely that someone or something is spoofing bingbot.

      I'd love to hear what you guys think!

      Dana

      1 Reply Last reply Reply Quote 1
      • LesleyPaone
        LesleyPaone last edited by

        I have recently had Bingbot crawl a site until it almost locked the database up, so it is possible. If you have doubts whether it is Bing bot or not, take to the logs and start extracting the ip addresses. You can verify them here, http://www.bing.com/webmaster/help/how-to-verify-bingbot-3905dc26

        danatanseo 1 Reply Last reply Reply Quote 2
        • danatanseo
          danatanseo @LesleyPaone last edited by

          Thanks Lesley. Yes, I agree. I think the only way we are going to get a definitive answer is to look at the logs. We are working on getting access.

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Should you use robots.txt for pages within your site which do not have high quality content or are not contributing a great deal so when Google crawls your site the best performing content has a higher chance of being indexed?
            Jacksons_Fencing
            Jacksons_Fencing
            0
            5
            44

          • Indexing problem a large site
            Andy-Halliday
            Andy-Halliday
            1
            6
            67

          • My site is not being regularly crawled?
            Everett
            Everett
            0
            5
            138

          • How to create site map for large site (ecommerce type) that has 1000's if not 100,000 of pages.
            Robin_Jennings
            Robin_Jennings
            0
            10
            9.5k

          • How Often is Site Crawled
            Matt-Williamson
            Matt-Williamson
            0
            5
            1.8k

          • One large site or a few microsites?
            duoweb
            duoweb
            0
            3
            509

          • Crawling a subfolder with a dev site
            KeriMorgret
            KeriMorgret
            0
            2
            359

          • Site Crawl
            Sean_Dawes
            Sean_Dawes
            0
            2
            414

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy