The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. Google only crawling a small percentage of the sitemap

    Google only crawling a small percentage of the sitemap

    Technical SEO Issues
    4 2 131
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • chrissmithps
      chrissmithps last edited by

      Hi,

      The company which I work for have developed a new website for a customer, there URL is https://www.wideformatsolutions.co.uk I've created a sitemap which has 25,555 URL's. I submitted this to Google around 4 weeks ago and the most crawls that have ever occurred has been 2,379.

      I've checked everything I can think of, including;

      • Speed of website
      • Canonical Links
      • 404 errors
      • Setting a preferred domain
      • Duplicate content
      • Robots Txt
      • .htaccess
      • Meta Tags

      I did read that Matt Cutts revealed in an interview with Eric Enge that the number of pages Google crawls is roughly proportional to your pagerank. But I'm sure it should crawl more than 2000 pages.

      The website is based on Opencart, if anyone has experienced anything like this I would love hear from you.

      1 Reply Last reply Reply Quote 0
      • LoganRay
        LoganRay last edited by

        Looks to me like Google can't properly access your XML sitemap. I tried to put it into 2 different validator tools and URI Valet and none of those tools were able to access it. It could be something with HTTPS. Did you recently switch the site over to secure?

        chrissmithps 1 Reply Last reply Reply Quote 3
        • chrissmithps
          chrissmithps @LoganRay last edited by

          I did have some issues with this when we first launched the site, I will try and look into it further now. The HTTPS certificate is fairly new.

          Thanks for commenting 🙂

          LoganRay 1 Reply Last reply Reply Quote 1
          • LoganRay
            LoganRay @chrissmithps last edited by

            No problem! I meant to mention this in my first comment, but I also noticed that there's no robots.txt file in place. That's obviously not going to help your indexation problem too much, but nonetheless something you should know about.

            1 Reply Last reply Reply Quote 1
            • 1 / 1
            • First post
              Last post
            • Desktop & Mobile XML Sitemap Submitted But Only Desktop Sitemap Indexed On Google Search Console
              bridget.randolph
              bridget.randolph
              0
              6
              1.6k

            • Google Crawling Issues! How Can I Get Google to Crawl My Website Regularly?
              libero_net
              libero_net
              1
              8
              142

            • Google News Sitemap
              0
              1
              102

            • I have 3500 pages crawled by Google, - why is SEOMOZ only able to crawl 400 of these ?
              Abe_Schmidt
              Abe_Schmidt
              0
              2
              95

            • Sitemap as Referrer in Crawl Error Report
              ChristinaRadisic
              ChristinaRadisic
              0
              3
              254

            • Sitemap and crawl impact
              TGOOSE
              TGOOSE
              0
              2
              90

            • CDN Being Crawled and Indexed by Google
              irvingw
              irvingw
              0
              2
              1.0k

            • My Website is not being crawled by Google
              MarieHaynes
              MarieHaynes
              0
              16
              2.0k

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy