Crawl Errors and Duplicate Content

[MyNet]

SEOmoz's crawl tool is telling me that I have duplicate content at "www.mydomain.com/pricing" and at "www.mydomain.com/pricing.aspx".  Do you think this is just a glitch in the crawl tool (because obviously these two URL's are the same page rather than two separate ones) or do you think this is actually an error I need to worry about?  Is so, how do I fix it?

[CodyWheeler]

The thing is, in the eyes of a crawler, they are different pages, just like http://domain.com and http://www.domain.com are the same page, but the crawlers will see them as different pages.

Are you using URL rewriting to get rid of the extension? If so I could see where this might cause a canonicalization issue if you don't tell the search engines which page you want to be ranked by using rel=canonical or redirecting from pricing.aspx to pricing.

Try using OSE on pricing and then pricing.html and see if you get different statistics.

[RyanKent]

This is not a glitch in the crawl tool. It is something that needs to be fixed.

As Cody suggested, search engines will not understand which URL is correct and any link credit can wind up being split.

Adding a canonical tag to your page will resolve the issue.

I would also examine the crawl report and look at the Referrer to determine if you have any links to the undesired page.

[MyNet]

Hmm, I don't understand this.  If a server can detect that these two URLs are the same why can't Google's billion dollar algorithm detect that these are the same?

[MyNet]

What page should I add the canonical tag to.  From my research a canonical tag in function operates in a similar way to the 301 redirect.  However I only have one content page even though it has two URL's.  Do I need to create literally two different versions of this content and put the canonical tag on the unwanted page?

[RyanKent]

You would add the canonical tag to your existing page.

You need to decide how you wish your page to be listed.

www.mydomain.com/pricing

www.mydomain.com/pricing.aspx

Those are two different URLs. They COULD lead to two different pages, but you are choosing to have them lead to the same page, which is a very standard practice. You need to let search engines know how you want the page to be listed. The URL without the .aspx extension is the friendlier URL. I would suggest choosing that one but it is up to you.

[duncan274]

This post is deleted!

[duncan274]

This post is deleted!

[RyanKent]

There are two aspects to the issue:

1. Resolve the cause of the problem. Crawl your site, locate any links to alternate URLs and change the links on your site to use the correct version of the URL.

2. Add a 301 redirect from /bad-url to /good-url. This will ensure any link juice to the bad urls is retained, along with providing a good user experience.