Submitting XML Sitemap for large website: how big?

[RG_SEO]

Hi there,

I’m currently researching how I can generate an XML sitemap for a large website we run. We think that Google is having problems indexing the URLs based on some of the messages we have been receiving in Webmaster tools, which also shows a large drop in the total number of indexed pages.

Content on this site can be accessed in two ways. On the home page, the content appears as a list of posts. Users can search for previous posts and can search all the way back to the first posts that were submitted.

Posts are also categorised using tags, and these tags can also currently be crawled by search engines. Users can then click on tags to see articles covering similar subjects. A post could have multiple tags (e.g. SEO, inbound marketing, Technical SEO) and so can be reached in multiple ways by users, creating a large number of URLs to index.

Finally, my questions are:

1. How big should a sitemap be? What proportion of the URLs of a website should it cover?
2. What are the best tools for creating the sitemaps of large websites?
3. How often should a sitemap be updated?

Thanks

[NHA_DistanceLearning]

Hi!

First off, since your content can be accessed in multiple ways, I'd make sure that you're applying means to indicate duplicate pages as such to search engines. Easy access to great content is fantastic, but you can devaluate your own pages a lot when you're not careful. If you're not using it yet, I recommend implementing the rel="canonical" tag in your website.

To answer your questions:

1. It should cover all URLs that want indexed. Ideally, that would be every URL
2. I'm not sure what 'the best' tools would be, but I used http://www.xml-sitemaps.com a lot a few years back. Their sitemaps are free up to 500 URLs. There are payment plans for bigger ones.
3. I wouldn't update an XML sitemap for every new page you make once a month. Instead, let the search engine find their own way in that case. Should your entire site structure change, an XML sitemap can be a great way to help search engine understand your new site setup better.

I hope this helps!

[MattAntonino]

I've created sitemaps with the paid version of Screaming Frog that were almost 80,000 pages. That's what I'd use.  No point asking what % unless you can't get it all.  If you're crawling Microsoft, break it up.  Otherwise, organize it if you can (category sitemap, month by month, something.)  or just make one big finger to Google type sitemap. lol

[RG_SEO]

Thanks for both your replies - I will check out the tools and recommendations you suggested.

I'm sure I remember somewhere reading a recommendation that it was only necessary to submit the basic site structure in a sitemap. It sounds like this is not the case and that a site map should , if possible, be comprehensive.

Would it be better to have a basic sitemap giving the main navigational URLs than having nothing at all?

[MattAntonino]

Yeah, it's better to have one than not - but I have always aimed to make it as complete as I can.  Why?  I'm not sure - mostly because I figure Google is GREAT at crawling my main structure - it's those far-reaching pages that I'm hoping they find in the sitemap.

[RG_SEO]

Thanks Matt, that's really useful