Duplicate Content Issue

[Ideas-Money-Art]

Why do URL with .html or index.php at the end are annoying to the search engine? I heard it can create some duplicate content but I have no idea why? Could someone explain me why is that so?

Thank you

[Naghirniac]

There is also the problem with http://www.example.com and http:/example.com - they also generate duplicate content.

What you should do?

First - solve the http://www.example vs http://example issue
Edit your .htaccess file (if you don't have it, create one). Inside this file you need to input:

RewriteEngine On RewriteCond %{HTTP_HOST} ^example.com RewriteRule (.*) http://www.example.com/$1 [R=301,L]
(where example.com is your website

Second - solve the index.html duplicate content

You need to include in the index.html file the follow metag inside the section:

This tag will tell google to forget the index.html and focus at www.example.com. So you will avoid the duplicate content without any problem

I hope I could help you.

[Vizergy]

Agreed

[Vahe.Arabian]

If there is the same page with the urls domain.com/seo & domain.com/seo.htm for this is what can be considered as duplicate content. With or without (better without) there should only be one version of the URL, to not split the link juice passing through. Hope this helps, Vahe

[blacey]

Using this as an example: If google showed duplicate content for http://www.domain.com/seo.html  and http://www.domain.com/seo then you would want a 301 .htaccess redirect so anyone accessing the http://www.domain.com/seo.htm version of the page is automatically sent to http://www.domain.com/seo.

You can use the following in your .htaccess file.

RewriteRule ^/seo.html$ http://www.domain.com/seo/? [R=301,NC,L]

The benefit to having no file extension (.html) on the end of your URLs is it allows you to change the underlying framework or your website or even the programming language without the need for adding redirects each time you change (provided the website structure remains the same)