Duplicate Content and URL Capitalization

[Jom]

I have multiple URLs that SEOMoz is reporting as duplicate content.  The reason is that there are characters in the URL that may, or may not, be capitalized depending on user input.

A couple examples are:

www.househitz.com/Pennsylvania/Houses-for-sale

www.househitz.com/Pennsylvania/houses-for-sale

www.househitz.com/Pennsylvania/Houses-for-rent

www.househitz.com/Pennsylvania/houses-for-rent

There are currently thousands of instances of this on the site.

Is this something I should spend effort to try and resolve (may not be minor effort), or should I just ignore it and move on?

[Marcus_Miller]

Hey Jom

Problem is, from a search engine perspective, those are four duplicate pages & from a linking perspective, they are four different pages that you could see your link popularity shared between. Neither of which is ideal.

I would certainly deal with this but it needn't be an arduous task.

1. Set up a rewrite rule to change all URLs to lowercase and 301 any non lowercase ones, something like this in your htaccess should do the job assuming you are using a LAMP environment.

RewriteEngine On
RewriteMap  lc int:tolower
RewriteCond %{REQUEST_URI} [A-Z]
RewriteRule (.*) ${lc:$1} [R=301,L]

2. Add an automated lowercase canonical to all of these pages so they canonicalise to the lowercase version.

3. Try to replace the links so they all use lowercase. If this is a dynamic site it should be easy but if not, you could still do a string replacement across multiple files. You could write a little script to automate this if it is a huge job from the sitemap (of lowercase URLs of course.

---

Certainly worth doing and should not be too difficult with a bit of smarts applied.

Hope this helps!
Marcus

[MarkLoud]

Hey Jom,

I must admit I am not sure on the level of urgency to sort this problem out but personally I like to keep the duplication of content to a minimum.

There are multiple ways to sort this out but the most straight forward would probably be to add a rel canonical tag to your web pages.

Here is a good post discussing the faceted issues you can get from e-commerce site, here is SEOMoz's canonicalization guide and here is another seomoz blog post about e-commerce sites and the use of the rel canonical tag.

Hope this helps

[Jom]

Hi Marcus and Mark,

Thanks for the response.  On creating the rel="canonical" statements.

That means that I will have thousands, perhaps hundreds of thousands (there are a lot of cities and zips in the US) of rel="canonical" statements on my site.

I thought I read on one of the blogs that too many canonical statements are bad practice.  The site is dynamic (Ruby on Rails), I can certainly make the change.  I would just like to be sure it's the wise thing to do.

[Jom]

Thanks, much!  I will read through these.

[Marcus_Miller]

Yeah, it is absolutely the right thing to do. You can force the URLs t be lower case in RoR as well if you don't want to do it in htaccess (i would do both).

You are simply saying:

• there are multiple versions of this page on different urls

• this is the main version of the page

301 them to lower case and canonicalise them and you are good to go!

Marcus

[Jom]

When you rewrite a URL that is already lower case  to lower case with a 301 response code, does it now return a 301?  Does that mean all pages on the site now return 301? Wouldn't that be bad?

Sorry if I'm being dense.  I understand enough about rewrite rules to be dangerous (sometimes, very dangerous).

Jom

[Jom]

Mark,

In the canonicalization guide link you sent me, there is a link to Matt Cutts' blog www.mattcutts.com/blog/seo-advice-url-canonicalization/ where he talks about it.  In that blog he posts:

Q: So when you say www vs. non-www, you’re talking about a type of canonicalization. Are there other ways that urls get canonicalized?
A: Yes, there can be a lot, but most people never notice (or need to notice) them. Search engines can do things like keeping or removing trailing slashes, trying to convert urls with upper case to lower case, or removing session IDs from bulletin board or other software (many bulletin board software packages will work fine if you omit the session ID).

This makes me think that doing a 301 redirect and a rel="canonical" for lower case is not needed.

I'm conflicted again.

[Marcus_Miller]

Hey Jom, you only rewrite the URL if it is not all lowercase, you can distinguish between lower and upper-case in your rewrites.