Duplicate content warning: Same page but different urls???

[ydf]

Hi guys i have a friend of mine who has a site i noticed once tested with moz that there are 80 duplicate content warnings, for instance

Page 1 is http://yourdigitalfile.com/signing-documents.html

the warning page is http://www.yourdigitalfile.com/signing-documents.html

another example

Page 1 http://www.yourdigitalfile.com/

same second page http://yourdigitalfile.com

i noticed that the whole website is like the nealry every page has another version in a different url?, any ideas why they dev would do this, also the pages that have received the warnings are not redirected to the newer pages you can go to either one???

thanks very much

[TimHolmes]

OK, so in this instance the only issue you have is that you need to choose your preferred start point - www or non www.

I would add a bit of code to your htaccess file to point to your preferred choice. I personally prefer a www. domain. Something like the below would work.

RewriteCond %{HTTP_HOST} ^example.com$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

As your site is already indexed I would also for the time being and as more of a safety measure add canonicals to the pages that point to the www. version of your site.

Also if you have  a Google Search Console account, you can select your prefered domain prefix in there. this will again help with your indexation.

Hopefully I have covered most things.

Cheers

Tim

[ydf]

Hey Tim

Thanks for your answer.  It's really weird, other than lazyness on the devs part not to remove old or previous versions of pages?, have you any idea why they would create multiple versions of the same page with different url's?? is there any legit reason like ones severs mobile or something??

Just wondering   thanks for replying

[TimHolmes]

I wouldn't say that they have created two pages, it is just that because you have two versions of the domain and not set a preferred version that you are getting it indexing twice. .HTaccess changes are under the hood of the website and could have simply been an oversight.

[ydf]

Oh ok

The only reason i was thinking it is duplicate content is the warnings i got on the moz crawl, see below.

75 Duplicate Page Content

6 4xx Client Error

5 Duplicate Page Title

44 Missing Meta Description Tag

5 Title Element is Too Short

I have found over 80 typos, grammatical errors, punctuation errors and incorrect information which was leading me to believe the quality of the work and their attention to detail was rather bad, which is why i thought this was a possibility.

Thanks again for your time its really appreciated

[AspenFasteners]

We have a similar problem, but much more complex to handle as we have a massive catalog of 80,000 products and growing.

The problem occurs legitimately because our catalog is so large that we offer different navigation paths to the same content.

http://www.aspenfasteners.com/Self-Tapping-Sheet-Metal-s/8314.htm

http://www.aspenfasteners.com/Self-Tapping-Sheet-Metal-s/8315.htm

(If you look at the "You are here" breadcrumb trail, you will see the subtle differences in the navigation paths, with 8314.htm, the user went through Home > Screws, with 8315.htm, via Home > Security Fasteners > Screws).

Our hosted web store does not offer us htaccess, so I am thinking of excluding the redundant navigation points via robots.txt.

My question: is there any reason NOT to do this?

[TimHolmes]

Hi there AspenFasteners, in this instance rather than a .HTAccess rule I would suggest applying a rel canonical tag which points to the page you deem as the original master source.

Using the robots to try and hide things could potentially cause you more issues as your categories may struggle to be indexed correctly.

[AspenFasteners]

Thanks Tim. Do you have any examples of what those problems might be? With such a large catalog managing those rel canonical tags will be difficult (I don't even know if the store allows them, it's a hosted store solution and little code customization is allowed).