The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Bingbot appears to be crawling a large site extremely frequently?

    Bingbot appears to be crawling a large site extremely frequently?

    Technical SEO Issues
    3 2 557
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • danatanseo
      danatanseo last edited by

      Hi All! What constitutes a normal crawl rate for daily bingbot server requests for large sites? Are any of you noticing spikes in Bingbot crawl activity?

      I did find a "mildly" useful thread at Black Hat World containing this quote: "The reason BingBot seems to be terrorizing your site is because of your site's architecture; it has to be misaligned. If you are like most people, you paid no attention to setting up your website to avoid this glitch. In the article referenced by Oxonbeef, the author's issue was that he was engaging in dynamic linking, which pretty much put the BingBot in a constant loop.

      You may have the same type or similar issue particularly if you set up a WP blog without setting the parameters for noindex from the get go."

      However, my gut instinct says this isn't it and that it's more likely that someone or something is spoofing bingbot.

      I'd love to hear what you guys think!

      Dana

      1 Reply Last reply Reply Quote 1
      • LesleyPaone
        LesleyPaone last edited by

        I have recently had Bingbot crawl a site until it almost locked the database up, so it is possible. If you have doubts whether it is Bing bot or not, take to the logs and start extracting the ip addresses. You can verify them here, http://www.bing.com/webmaster/help/how-to-verify-bingbot-3905dc26

        danatanseo 1 Reply Last reply Reply Quote 2
        • danatanseo
          danatanseo @LesleyPaone last edited by

          Thanks Lesley. Yes, I agree. I think the only way we are going to get a definitive answer is to look at the logs. We are working on getting access.

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Help Crawl friendliness for large site
            TakeshiYoung
            TakeshiYoung
            0
            4
            144

          • How to create site map for large site (ecommerce type) that has 1000's if not 100,000 of pages.
            Robin_Jennings
            Robin_Jennings
            0
            10
            9.5k

          • Seomoz Can not Crawl My Site
            AndreVanKets
            AndreVanKets
            0
            2
            112

          • Can anyone help me understand why google is "Not Selecting" a large number of my webpages to include when crawling my site.
            Jdubin
            Jdubin
            0
            4
            356

          • Have a client that migrated their site; went live with noindex/nofollow and for last two SEOMoz crawls only getting one page crawled. In contrast, G.A. is crawling all pages. Just wait?
            Nobody1560986989723
            Nobody1560986989723
            0
            5
            422

          • One large site or a few microsites?
            duoweb
            duoweb
            0
            3
            509

          • How do you diagnose if on your site is only 50% crawled?
            Nightwing
            Nightwing
            0
            3
            326

          • Recently revamped site structure - now not even ranking for brand name, but lots of content - what happened? (Yup, the site has been crawled a few times since) Any ideas? Did I make a classic mistake? Any advise appreciated :)
            AlanBleiweiss
            AlanBleiweiss
            0
            9
            844

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy