Using Meta Header vs Robots.txt

[AlanMosley]

I'm not sure you have a problem, why not let them all get indexed?

[evan89]

It is a problem in the SERPS because if I run a query for the brand, I can see faceted variations of that brand (say "brand" "blue") is ranking right below, but neither of them are ranking on the first page. I won't NOINDEX all pages, just those that don't provide value for customers searching, and those that are competing with competitive terms that are causing the preferred page to rank lower.

It was brought to my attention through Moz analytics, and once I began to investigate it further, I found many sources mentioning that this is very common for e-commerce. Common practice is robots.txt and a plugin, but we are using a different plugin. So, for this reason, I am trying to figure out if NOINDEX meta headers are a good option.

Does that make sense?

[CraigBradford]

Hi Even, this is quite a common problem. There are a couple of things to consider when deciding if Noindex is the solution rather than robots.txt.

Unless there is a reason the pages need to be crawled (like there are pages on the site that are only linked to from those pages) I would use robots.txt. Noindex doesn't stop search engines crawling those pages, only from putting them in the index. So in theory, search engines could spend all there time crawling pages that you don't want to be in the index.

Here's what I'd do:

Decide on a reasonable number of facets, for example, if you're selling TVs people might search for:

1. Sony TV (Brand search)
2. 50 inch sony tv (size + brand)
3. Sony 50 inch HD TV (brand + size + specification)

But past 3 facets tends to get very little search volume (do keyword research for your own market)

In this case I'd create a rule that appends something to the URL after 3 facets hat would make it easy to block in robots.txt. For example I might make my structure:

1. example.com/tv/sony
2. example.com/tv/sony/50
3. example.com/tv/sony/50/HD

But as soon as I add a 4th facet, for example 'colour'- I add in the filter subfolder

• example.com**/filter/**tv/sony/50/HD/white

I can then easily block all these pages in robots.txt using:

Disallow: /filter/

I hope this helps.

[AlanMosley]

The problem with robots text is that any link pointing to a no-indexed page is passing link juice that will never be returned, it is wasted. robots.txt is the last resort, IMO its should never be used.

[CraigBradford]

Hi Alan, I understand that, but the problem Evan is describing seems to be related to duplicate content and crawl allowance. There's no perfect answer but in my experience the types of pages that Evan is describing aren't often linked to. Taking that into consideration, IMO robots.txt is the correct solution.

[AlanMosley]

they will be linked to by internal links,

There is no penalty for have duplicates of your own content, but having links pouring away link juice is a self imposed penalty.

[evan89]

Hey Craig,

Thanks for your response. This is the common answer that I have found. Here is the challenge I am having (I will use your example above):

Let's say that example.com/tv/sony is the main category page for this brand, but I only carry a few Sony tvs. Therefore, the only difference between that page and this page: example.com/tv/sony/50 is a category description that disappears when further facets are chosen.

When I search in the SERPS for "Sony TVs", rather than ranking well for one of these pages, both rank moderately well, but not well enough for first page results, and I would think this is confusing to customers as well to find two very closely related pages side by side.

So, while I agree that robots.txt is a tool that I can apply for limiting search engines from getting dizzy with the facets by limiting them to (say) 4, is NOINDEX the best solution for controlling duplicate content issues that are not that deep, and more case-by-case?

One more thing I might add is that these issues don't happen site-wide. If I carry many products from Samsung, than example.com/tv/samsung and example.com/tv/samsung/50 and even example.com/tv/samsung/50/HD will produce very different results. The issue usually occurs where there are few products for a brand, and they filter the same way with many facets.

Does that make sense? I agree with you whole heartedly, I am just trying to figure out how to deal with the shallow duplicate issues.

Cheers,

[AlanMosley]

This sounds like a job for a canonical tag.

[evan89]

As mentioned initially, the CMS doesn't allow me to edit canonicals for individual pages that are created via facets. The other question I had about canonicals is that a rel canonical is meant to help bots understand that different variations of the same page are, in fact, the same page: example.com = example.com/. But, for the user (which ultimately bots care about), example.com/sony/50 may not always be the same as example.com/sony.

Anyways, that is a little beside the point. I have visited the options of canonicals, but I am not sure it can be done.

[evan89]

Hey Craig,

I agree with you regarding the robots.txt, however, how does one isolate parameters that are commonly used within product names, thus being the the product url as well. We are using a plugin the makes the urls more user friendly, but it makes it tough to isolate "small" or "blue" because the parameters don't include a "?sort=" or "color=" prefix anymore.

This is why I am considering using the meta header in order to control help with the issues of the duplicate content and crawl allowance?

Since I can edit the meta headers on a variety of pages, is it a viable option to use NOINDEX,FOLLOW?

[evan89]

"there is no penalty for have duplicates of your own content"

Alan,

I must respectfully disagree with this statement. Perhaps google will not penalize you directly, but it is easy to self-canabalize key terms if one has many facets that only differ slightly. I have seen this on a site where they don't rank on the first page, but they have 3-4 pages on the second page or SERPs. This is the exact issue that I am trying to resolve.

Evan

ps. sorry I hit the wrong button, but you got a good answer out of it