Sitemap.xml - autogenerated by CMS is full of crud

[k3nn3dy3]

Hi all,

hope you can help.

the Magento ecommerce system I'm working with autogenerates sitemap.xml - it's well formed with priority and frequency parameters.

However, it has generated lots of URLs that are pointing to broken pages returning fatal erros, duplicate URLs (not canonicals), 404s etc

I'm thinking of hand creating sitemap.xml - the site has around 50 main pages including products and categories, and I can get the main page URLs listed by screaming frog or xenu.

Then I'll have to get into the hand editing the crud pages with noindex, and useful duplicates with canonicals.

Is this the way to go or is there another solution

thanks in advance for any advice

[KaneJamison]

If it's returning 404 pages, that sounds like a dated sitemap. Have you activated the cron service?

See the "Refreshing Sitemaps at Regular Intervals" section of this page if not:

Magento can be set up to automatically refresh Google Sitemaps at regular intervals. This function is configured in Admin > System > Configuration > Google Sitemap.

To use Magento’s automatic generation of Google Sitemaps, you must activate the Magento Cron service.

If you do have that setup, and you're certain it's working correctly, then I would turn to the forums at MagentoCommerce.com - you're going to get a lot faster answer there since everyone is familiar with that exact platform.

[k3nn3dy3]

Hi Kane,

the sitemap is new - it's just that Magento create lots of duplicate files on the fly & it's not putting the canonical URLs in the sitemap etc.

I just wondered whether its worth hand creating a sitemap.xml containing the content pages (60 or 70 of them) for this relatively small site, or not worry too much about the sitemap, the site is pretty well indexed by google already

I'll head over to the Magento forums again to see if I can find more info

many thanks for you help

[KaneJamison]

If the cron is working then I would personally turn to the other forum to see if anyone knows a way to rope those messy URLs in and get them under control. I try to avoid manually generating and updating sitemaps whenever I can, because it's a hassle on a small site, not to mention the trouble on an ecommerce site.

If your site is going to stay that small, then a manual sitemap might be less of a headache for you than customizing Magento.

I would worry about keeping a clean sitemap. If the search engines learn that you keep a messy sitemap, they will rely on it less and less. 404 & 500 codes especially, but also redirects and perhaps duplicate content.

For Further Reading:

Google Sitemaps Ask For Clean URLs - http://www.johnfdoherty.com/google-sitemaps-ask-for-clean-urls/