The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Search Engine Trends
    4. Is googlebot the slowest bot?

    Is googlebot the slowest bot?

    Search Engine Trends
    4 2 67
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • loopyal
      loopyal last edited by

      This morning, I wrote a breaking news story about a "Wolf of Wall Street"

      It was published at 12:05:49

      Googlebot, which used to be on my site within a minute or less, didn't bother to visit for 53 minutes. And now, 32 minutes later, even though it has been crawled, this story doesn't even show up in google search.

      Except that it is in the top 10 stories today, at #2, so the headline appears in every page on the site, so every page that has been crawled today, around 10 minutes after it was published, contains that text, so they show up. EINnews, which also crawls our pages is listed for the headline text.

      Finally, the page turns up in search results 4 hours later, and says that it is 4 hours old.

      Does anyone else see this slow motion mode?

      If you do see this, what is wrong with the site that causes this recalcitrant behavior?

      The headline of the story is "A 'Wolf of Wall Street' Raided By FBI In Florida"

      and the link is http://shar.es/1bW5Sw

      1 Reply Last reply Reply Quote 0
      • RyanPurkey
        RyanPurkey last edited by

        Hi Alan,
        Are you still pushing Google News tagged XML sitemaps when publishing articles as well?  Looking at the ones currently on your site I don't see any new ones referenced since October 2014. And it looks like there's a lot of current mapping that could be updated.

        In general the site seems a little low in the loop of the major news cycle and would have a lower crawl/index priority on big stories behind the CNNs, Foxs, and Yahoos of the world.

        It also doesn't seem to be in the Google News index: https://encrypted.google.com/search?hl=en&q=site%3Anewsblaze.com#hl=en&tbm=nws&q=wolf+of+wall+street+site:newsblaze.com

        Google's Guideline to Google News Publisher inclusion is straightforward, and fairly thorough. https://support.google.com/news/publisher/answer/40787?hl=en  If you get included via those means, you should see your news articles appearing very quickly.

        1 Reply Last reply Reply Quote 1
        • loopyal
          loopyal last edited by

          Thank you for looking Ryan.

          Google ignores out news map because they removed us from new for a reason they didn't disclose, about 2 years ago.

          So I haven't been checking on that map.

          I've been trying to find time to change over from my custom-built CMS to Wordpress, and thought I'd reapply after I did that, but I'm 6 months behind my schedule to get that done. (had problems with the page design and the data conversion)

          Yes, we're much smaller than the others, but 4hours for a page to show up in the index must mean something else is going on, and I can't work out what that could be.

          I'll see if I can get my redesign back on track, and that will make the site more mobile-friendly.

          Have you seen anything like that 4 hour gap before? I will track the next few stories I publish too, and report back.

          1 Reply Last reply Reply Quote 0
          • RyanPurkey
            RyanPurkey last edited by

            Your gap will disappear if you get back into the News index. Best of luck!

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • When block bots using robots.txt vs meta tag "no index"?
              gertseoleverage
              gertseoleverage
              0
              5
              79

            • Bot Crawling issues
              SubhaVaastu
              SubhaVaastu
              0
              5
              78

            • Log File Analyzer Only Showing Spoofed Bots and No Verified Bots
              screamingfrog
              screamingfrog
              0
              2
              161

            • Googlebot soon to be executing javascript - Should I change my robots.txt?
              danatanseo
              danatanseo
              0
              3
              504

            • Why do we have so many pages scanned by bots (over 250,000) and our biggest competitors have about 70,000? Seems like something is very wrong.
              danatanseo
              danatanseo
              0
              4
              166

            • Struggling with Google Bot Blocks - Please help!
              Devanur-Rafi
              Devanur-Rafi
              0
              11
              213

            • Search bots that use referrers?
              0
              1
              573

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy