The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Can't generate a sitemap with all my pages

    Can't generate a sitemap with all my pages

    Intermediate & Advanced SEO
    5 4 663
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Banknotes
      Banknotes last edited by

      I am trying to generate a site map for my site nationalcurrencyvalues.com but all the tools I have tried don't get all my 70000 html pages...   I have found that the one at check-domains.com crawls all my pages but when it writes the xml file most of them are gone... seemingly randomly.

      I have used this same site before and it worked without a problem.  Can anyone help me understand why this is or point me to a utility that will map all of the pages?

      Kindly,

      Greg

      1 Reply Last reply Reply Quote 0
      • LoganRay
        LoganRay last edited by

        Greg,

        Have you tried creating multiple XML sitemaps by section of the site, like by folder or by product detail pages? 70,000 is a huge amount of URLs and even if you could get them all on one sitemap, I wouldn't recommend it. Nesting sitemaps into an index sitemap can help Google understand your site structure and make it easier for you to troubleshoot indexing problems should they arise.

        1 Reply Last reply Reply Quote 3
        • Tylerj
          Tylerj last edited by

          Looking at your site how sure are you that you need 70,000 pages?

          For the sitemap I would stop trying to use a website and do it yourself. It looks like you are running IIS. They have a sitemap generator that you can install on a server easily and run it there. It looks like you have GoDaddy, they catch a lot of crap but I have always found their technical support to be top notch. If you can't figure out how to do it on the server I would give them a call.

          1 Reply Last reply Reply Quote 1
          • dohertyjf
            dohertyjf last edited by

            I definitely agree with Logan. The max for an XML sitemap for Search Console is 50,000 URLs, so you won't be able to fit all of yours into one.

            That being the case, divide them into different sitemaps by category or type, then list all of those in one directory sitemap and submit that. Now you can see indexation by page type on your website.

            Finally, I have to ask why you are doing this with a third party tool and creating a static sitemap as opposed to creating a dynamic one that can update automatically when you publish new content? If your site is static and you're not creating new pages, then your approach might be ok, but otherwise I'd recommend investigating how you build a dynamic XML sitemap that updates with new content.

            Cheers!

            1 Reply Last reply Reply Quote 1
            • Banknotes
              Banknotes last edited by

              Thank you all for the responses... I found them all helpful.  I will look into creating my own sitemap with the IIS tool.

              I can't help the 70k pages but the URLS are totally static.  I guess I can make a site map for all the aspx pages and then other one for all the lowest level .html pages.

              Thanks everyone!

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              • First post
                Last post
              • Crawl and Indexation Error - Googlebot can't/doesn't access specific folders on microsites
                Everett
                Everett
                0
                2
                87

              • Why some websites can rank the keywords they don't have in the page?
                TheSymmetran
                TheSymmetran
                0
                6
                115

              • What can you do when Google can't decide which of two pages is the better search result
                David-Kley
                David-Kley
                0
                3
                85

              • Can't get page moving!
                MattAntonino
                MattAntonino
                0
                3
                95

              • I've seen and heard alot about city-specific landing pages for businesses with multiple locations, but what about city-specific landing pages for cities nearby that you aren't actually located in? Is it ok to create landing pages for nearby cities?
                AlanMosley
                AlanMosley
                1
                3
                1.9k

              • How can Google index a page that it can't crawl completely?
                OlegKorneitchouk
                OlegKorneitchouk
                0
                4
                75

              • If I can't list my canonical URLs in my sitemap, will I be creating duplicate content?
                emilyburns
                emilyburns
                0
                3
                2.2k

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy