How to create site map for large site (ecommerce type) that has 1000's if not 100,000 of pages.

[BestRide]

I know this is kind of a newbie question but I am having an amazing amount of trouble creating a sitemap for our site Bestride.com.  We just did a complete redesign (look and feel, functionality, the works) and now I am trying to create a site map.  Most of the generators I have used "break" after reaching some number of pages.  I am at a loss as to how to create the sitemap.  Any help would be greatly appreciated!

Thanks

[Chris.Menke]

You can use screamingfrog to create your sitemap.  You just need to license it for crawl more than 500 URI.

[Red_educativa]

Hi Kristin,

Each sitemap.xml can support maximum 50.000 URLs. So, If you have a site with more than 100K, It'd be better to create 2 or 3 o 4 etc sitemaps.xml in order to contain all URLs. Hope it is useful.

Kind regards!

Francesca

[Chris.Menke]

Of course, you can also use the moz's crawl test report at http://pro.moz.com/tools/crawl-test

[LesleyPaone]

Are you using a custom platform or an off the shelf e-commerce package? Most off the shelf packages actually have a module that can create a site map and a lot have it where you can cron it too.

[BestRide]

Thanks for the feedback!

I will look into screamingfrog for sure.

@Lesley - we are using a custom platform (in house) so we don't have that functionality.  The issue is that we have a lot of inventory (millions) of cars.  We have built (and are releasing new functionality today) to provide internal links so that Google can crawl all the inventory easily (users can too :).  My question about sitemaps has boiled down to this: Do we need to build the sitemap to include every single page (all the inventory) or do we provide a "map" so that google can find the top pages and then crawl the inventory from there.  Again the site is bestride.com.  If anyone wants to take a look at the site, that would be fantastic!

Thanks

[Chris.Menke]

Typically, a sitemap is going to include every page on the site. As Francesca said, each sitemap can be up to 50K urls and if you need multiple sitemaps then you create a sitemap index that points to the rest of the sitemaps.

https://support.google.com/webmasters/answer/183668?hl=en

[BestRide]

That's a great help Chris, thank you!  And thanks to all for your help!

[LesleyPaone]

The easiest thing i can think of is to write a script that works with your dispatcher to create a site map. The format I would use is add the page and all of the "product images" on the page to the map and move to the next. At the same time I would use an auto increment variable to keep track of how many lines you have written. When you get around 50k, write out the name of the next site map file that the program will create and have them chained together this way.

[Robin_Jennings]

I agree with Chris. With such large websites it would be advisable having a sitemap index and then splitting the index into various individual indexes such as Pages, Products, Categories, images, media, tags etc.