Duplicate content

[Paul78]

I have just ran a report in seomoz on my domain and has noticed that there are duplicate content issues, the issues are:

www.domainname/directory-name/

www.domainname/directory-name/index.php

All my internal links and external links point to the first domain, as i prefer this style as it looks clear & concise, however doing this has created duplicate content as within the site itself i have an index.php page inside this /directory-name/ to show the page.

Could anyone give me some advice on what i should do please?

Kind Regards

[AdeLewis]

Hey Gary.

Here's the solution that I use.

All my sites are hosted on a linux server so this won't be relevant if your site is hosted on a windows server.

1. create/modify your .htaccess file in your site's root directory.

2. Add the following code to the top of the file:-

RewriteEngine On

RewriteBase /
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /index.php\ HTTP/
RewriteRule ^index.php$ http://www.yourdomain.com/ [R=301,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

RewriteCond %{HTTP_HOST} ^yourdomain.com [NC]
RewriteRule ^(.*)$ http://www.yourdomain.com/$1 [L,R=301]

This will ensure that any requests sent to http://yourdomain.com are redirected to http://www.yourdomain.com and that the index.php part of the url is removed.

If you need more help on creating or modifying your .htaccess file then you can find more info here - http://httpd.apache.org/docs/1.3/howto/htaccess.html

All the best.

Ade.

[Paul78]

Hi Ade,

Thanks for the speedy reply.

I have now implemented this and works fantastic on the http://www.mydomain.com/

Thank you very much.

There is another issue however, i hope i can make sense here, here goes:

seomoz tool gives me back duplicate content on both these URL's

http://www.mydomain.com/football-teams/

http://www.mydomain.com/football-teams/index.php

I want to use http://www.mydomain.com/football-teams/  as this just look nice & clean.

What would be best practice to fix this issue?

Kind Regards

[AdeLewis]

Hi Gary.

Have you tried visiting the url http://www.mydomain.com/football-teams/index.php to see if it now resolves to http://www.mydomain.com/football-teams/ ?

If it does then the issue is fixed, the next time SEOMoz crawls your site the error will dissapear.

Cheers.

Ade.

[Paul78]

Hi Ade,

Yes, i tested http://www.mydomain.com/football-teams//index.php however it did not resolve to http://www.mydomain.com/football-teams/

Any ideas?

[AdeLewis]

In that case you can just add a 301 redirect in to your .htaccess file below the code you added earlier.

redirect 301 /football-teams/index.php http://www.mydomain.com/football-teams/

[Paul78]

Do you think that's practical to do that?

As i will need to do a 301 on literally every page if i don't want to show the /index.php

Is this what seomoz.org website does? for example:

http://www.seomoz.org/community

[AdeLewis]

Ahhhh. No definitely not practical, I thought that it was just the one url.

Are you using a content management system for your site such as Joomla?

[Paul78]

We are using a CMS, its called WebEdition, is there a technical question i should ask them in what i need to do?

Kind Regards

[AdeLewis]

Hi Gary,

That one is a bit beyond me I'm afraid and I am not familiar with WebEdition at all.

With most CMS there are normally either built-in or add-on extensions to help with re-writing your urls but you need to be really careful that you don't end up with a completely new set of urls that don't match either of your originals.

A .htaccess rewrite rule may be your best option but I don't know what the coding for it would be.

[Paul78]

Hi Ade,

Thanks for all your help.

I will post a new question on the Q&A Forum regarding the .htaccess rule.

Kind Regards

[JaspalX]

Hi Gary,

Here's some code from an htaccess file I've used before that solves the issue you've got with index.php at the end of all your urls:

#remove /index.php and ensure admin works okay

RewriteCond %{REQUEST_URI} !^/administrator

RewriteCond %{THE_REQUEST} ^.*/index.php\ HTTP/

RewriteRule ^(.*)index.php$ /$1 [R=301,L]

notice the line that contains ^/administrator , in Joomla, admin login is usuall on http://site.com/administrator/index.php

so, removing the index.php from the admin url would prevent any access to the admin screens! If your cms has a similar url, be sure to replace 'administrator' with the relevant url.