The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Baidu Spider appearing on robots.txt

    Baidu Spider appearing on robots.txt

    Intermediate & Advanced SEO
    3 2 242
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • IceIcebaby
      IceIcebaby last edited by

      Hi, I'm not too sure what to do about this or what to think of it.

      This magically appeared in my companies robots.txt file (literally magically appeared/text is below)

      User-agent: Baiduspider
      User-agent: Baiduspider-video
      User-agent: Baiduspider-image
      Disallow: /

      I know that Baidu is the Google of China, but I'm not sure why this would appear in our robots.txt all of a sudden. Should I be worried about a hack? Also, would I want to disallow Baidu from crawling my companies website?

      Thanks for your help,
      -Reed

      1 Reply Last reply Reply Quote 0
      • Travis_Bailey
        Travis_Bailey last edited by

        There's a possibility someone in your company saw suspicious traffic from an actor spoofing the Baidu user agent. It can get so aggressive that it will eventually bog down your response time through sheer number of requests. But the problem is that same actor, or someone else with malicious intent can simply spoof another user agent or IP.

        But the main problem is, the site is straight e-commerce. It could get international business, so why take such a ham fist approach? Even if blocking Baidu gave the desired result, the dev/admin would still have to block individual IP blocks as they come in. It would make more sense to invest in server resources so it can handle the load, or look into DDos Mitigation.

        So yeah, it's strange. Though it's more likely a lack of understanding than anything malicious.

        1 Reply Last reply Reply Quote 1
        • IceIcebaby
          IceIcebaby last edited by

          Thanks for your help Travis, that was a really solid answer.

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          • Is robots met tag a more reliable than robots.txt at preventing indexing by Google?
            Bobbi_Tschumper
            Bobbi_Tschumper
            1
            7
            3.0k

          • Robots.txt advice
            Martijn_Scheijbeler
            Martijn_Scheijbeler
            0
            3
            105

          • Robots.txt Allowed
            GlobeRunner
            GlobeRunner
            0
            4
            118

          • Have a Robots.txt Issue
            MattRoney
            MattRoney
            0
            5
            226

          • Robots.txt assistance
            theLotter
            theLotter
            0
            9
            280

          • Robot.txt File Not Appearing, but seems to be working?
            Vizergy
            Vizergy
            0
            4
            91

          • Robot.txt error
            Rubix
            Rubix
            0
            13
            203

          • Robots.txt Question
            Dr-Pete
            Dr-Pete
            0
            4
            231

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy