The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Search Engine Trends
    4. Reasons for a sharp decline in pages crawled

    Reasons for a sharp decline in pages crawled

    Search Engine Trends
    4 3 386
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Symmetri
      Symmetri last edited by

      Hello!

      I have a site I've been tracking using Moz since July. The site is mainly stagnant with some on page content updates. Starting the first week of December, Moz crawler diagnostics showed that the number of pages crawled decreased from 300 to 100 in a week.

      So did the number of errors through. So crawler issues went from 275 to 50 and total pages crawled went from 190 to 125 in a week and this number has stayed the same for the last 5 weeks.

      Are the drops a red flag? Or is it ok since errors decreased also? Has anyone else experienced this and found an issue?

      FYI: sitemap exists and is submitted via webmaster tools. GWT shows no crawler errors nor blocked URLs.

      1 Reply Last reply Reply Quote 0
      • MickEdwards
        MickEdwards last edited by

        How many URLs are indexed in Google if you use site:yourdomain.com  Has that figure dropped too?

        Have you got anything in your robots.txt that could be blocking?

        Symmetri 1 Reply Last reply Reply Quote 1
        • KeriMorgret
          KeriMorgret last edited by

          If you fixed a problem, such as duplicate content, that would mean that we're showing fewer errors and crawling fewer pages, since that problem is fixed. Might that be the case?

          1 Reply Last reply Reply Quote 3
          • Symmetri
            Symmetri @MickEdwards last edited by

            Google is indexing just over 80 URLs, although about 40% of them are developer test URLs (they lead to live pages of the site though). Nothing in robots.txt. No errors.

            The Google bot is still crawling, but it's crawling half the pages. What would make it decrease in page crawls? I'm working if there is a broken link or something on the home page that's pointing away from the site... although it's unlikely, I'll check....

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            • AMP Pages need to be exactly similar to the main page?
              effectdigital
              effectdigital
              0
              2
              154

            • Rel canonical on other page instead of duplicate page. How Google responds?
              vtmoz
              vtmoz
              0
              6
              90

            • Lots of dublicate titles and pages on search page
              webtheoria.com
              webtheoria.com
              0
              6
              126

            • Landing Pages are Constantly Fluctuating in Rankings - Is There a Reason Why?
              MonicaOConnor
              MonicaOConnor
              0
              3
              136

            • Should my canonical tags point to the category page or the filter result page?
              KeriMorgret
              KeriMorgret
              0
              5
              4.1k

            • Privacy page ranking above home page in serps
              edwardrj
              edwardrj
              0
              8
              472

            • Is my page footer the reason keyword rankings have dropped?
              RichardTaylor
              RichardTaylor
              0
              4
              467

            • Server Down for Few Hours went from Page 1 to Page 6?
              webdesigncwd
              webdesigncwd
              0
              9
              594

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy