An affiliate website uses datafeeds and around 65.000 products are deleted in the new feeds. What are the best practises to do with the product pages? 404 ALL pages, 301 Redirect to the upper catagory?

[Zanox]

Note: All product pages are on INDEX FOLLOW.

Right now this is happening with the deleted productpages:

1. When a product is removed from the new datafeed the pages stay online and are showing simliar products for 3 months. The productpages are removed from the categorie pages but not from the sitemap!

2. Pages receiving more than 3 hits after the first 3 months keep on existing and also in the sitemaps. These pages are not shown in the categories.

3. Pages from deleted datafeeds that receive 2 hits or less, are getting a 301 redirect to the upper categorie for again 3 months

4. Afther the last 3 months all 301 redirects are getting a customized 404 page with similar products.

Any suggestions of Comments about this structure?

Issues to think about:
- The amount of 404 pages Google is warning about in GWT
- Right now all productpages are indexed
- Use as much value as possible in the right way from all pages
- Usability for the visitor

Extra info about the near future: Beceause of the duplicate content issue with datafeeds we are going to put all product pages on NOINDEX, FOLLOW and focus only on category and subcategory pages.

[AlanMosley]

A few points, I would not put al these pages in the sitemap.xml, if you are a talking about a html sitemap visually showing on your site, then I would not have any links on y site that do not go directly to a page, don't want redirects from internal links, and you definitely don't want 404s.
You say you have a customized 404 page. does this page return a 404 status code? if not you will be getting soft 404 problems. Also a 404 page really should not have any images js or css files on it, especially if they are changing. if you get a broken link on your 404 page, such as a missing image, you will get a internal loop, and search engines really don't like that.

[Zanox]

Hi Alan,

Thanks for your reply!

Only the most impotant product pages are shown in the xml sitemap. Not in the HTML sitemap.
If the pages are live, than they stay in the sitemap. Off course when the pages are offline there is no sign of these pages anywere so not in the sitemaps and not in the website categories.

The custom 404 returns indeed a 404 status code, so we don't got soft 404 problems

We also don't have any broken links on the website beceause links of deleted products are not shown

Thats right! You don't want to end up in a 404 loop! Crawlers will instantly hate your website...
But we really want to show some links on the custom 404 to help the visitor searching what he/she's looking for.
Just have to make sure that the links are working right

_One question: You said "no redirect from internal links" but it's more about redirect from internal pages beceause all the links to productpages that are not exisiting anymore are getting removed.... Do you agree overall with the structure of 3 months redirecting and than the 404 page? _

_Note: the 3 months redirecting is important to not lose any visitors from organic rankings. It takes Google time to de-index the pages. _

Thanks in Advanced!

[AlanMosley]

OK I understand a bit better now, yes I agree with what you are doing. All makes sense

[Everett]

Also see answers on Wesley's other question thread here:
Should we noindex 65,000 product pages all at once or in steps? (paraphrasing)