Homepage duplicated content - 301 redirection - "Rel="canonical" - Headache

[mdmoz]

Hello Moz community,

I already have 301 redirection in my .htaccess , just like this:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^mysite.com$
RewriteRule ^(.*)$ "http://www.mysite.com/$1" [R=301,L]

Mozpro shows that I have duplicated content for www.mysite.com & www.mysite.com\index.html

I am a little lost, there is " 301 Redirect" and "Rel="canonical", moz replied me back that I need to use "Rel="canonical", but can one use both together?

i also read on forums, if I 301 redirect index to /, there will be infinite loop and such.

Another story, I am also getting IP Canonicalization error when I checked my SEO performance:

Your site's IP ...* does not redirect to your site's domain name. This could cause duplicate content problems if a search engine indexes your site under both its IP and domain name.**

Which adds to my headache. What is this one all about?

There is no single thread or page here, to clear all this confusion and give a straight solution, but rather each page or thread says do a different thing.

I am lost, so somebody help!

[BlueprintMarketing]

Hi,

I am going to try to make this as easy as possible before look by pasting your questions in italics and answering and non-italics.

I believe that you are new server setup if that is the case is in need of some basic setup

in addition to  correctly using the 301 redirect as well as the canonical tag to fix your issues.

I am a little lost, there is " 301 Redirect" and "Rel="canonical", moz replied me back that I need to use "Rel="canonical", but can one use both together?

i also read on forums, if I 301 redirect index to /, there will be infinite loop and such.

this is very important for you to read

http://moz.com/blog/apache-redirect-an-index-file-to-your-domain-without-looping

This is also an excellent reference

http://moz.com/learn/seo/canonicalization

** I cited this to save you time from the page below **

https://support.google.com/webmasters/answer/139066?hl=en

"Use 301 redirects for URLs that are not canonical

Suppose your page can be reached in multiple ways:

• http://example.com/home
• http://home.example.com
• http://www.example.com

It's a good idea to pick one of those URLs as your preferred (canonical) destination, and use 301 redirects to send traffic from the other URLs to your preferred URL. A server-side 301 redirect is the best way to ensure that users and search engines are directed to the correct page. The 301 status code means that a page has permanently moved to a new location."

http://moz.com/blog/canonical-url-tag-the-most-important-advancement-in-seo-practices-since-sitemaps

http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html

You want to place the canonical tag on the URL you wish to have search engines recognize as your home URL

Yes they can be used together if they are both pointing to the same place it is not that they have an issue.

The canonical tag tells Google this is the page I want whichever the canonical tag is pointing to.

Mozpro shows that I have duplicated content for www.mysite.com & www.mysite.com\index.htm

The correct Regex to accomplish that  just removed removed the index HTML file on your homepage

<code class="htaccess" title="in your .htaccess file">RewriteEngine On
RewriteRule ^index\.html$ / [R=301,L]</code>

this would remove it from your entire site

<code class="htaccess" title="in your .htaccess file">RewriteEngine On
RewriteRule ^index\.html$ / [R=301,L]
RewriteRule ^(.*)/index\.html$ /$1/ [R=301,L]</code>

This is exactly what I think you need to set it up correctly

In an existing .htaccess file or a new .htaccess file add the following code:

http://www.smartlabsoftware.com/howto/redirect-index-page.htm

Options +FollowSymLinks

RewriteEngine On

redirect index.htm and index.html to / (do this before non-www to www)

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.index.html?\ HTTP/
RewriteRule ^(.)index.html?$ http://www.mydomain.com/$1 [R=301,L]

redirect mydomain.com -> www.mydomain.com

RewriteCond %{HTTP_HOST} ^mydomain.com$ [NC]
RewriteRule ^(.*)$ http://www.mydomain.com/$1 [R=301,L]

Depending on entirely with the server you are currently running this could be of use as well.

http://css-tricks.com/how-to-redirect-indexhtml-to-indexphp/

.htaccess ReWrite

**`RewriteEngine On
RewriteBase /
RewriteRule ^index\.html?$ / [NC,R,L]`** 

Your site's IP ..*. does not redirect to your site's domain name. This could cause duplicate content problems if a search engine indexes your site under both its IP and domain name.

Which adds to my headache. What is this one all about?

you need to set your host file and server IP catch all it is illustrated beautifully in this

here is the short version

when you are setting up your server rather at your hosting company or yourself you need to set what is called a "catch all" this prevents the IP address acting like a domain.

As well as preventing people from just using their DNS to point their domain to your site and have it resolved. It is an extremely fundamental and important part of setting up a server correctly. You want to be certain that your catch all

host name being set up in a local database this Should be done by your hosting company if there managing the server however if it is a non managed server I recommend you follow the steps below.

# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1       localhost

#Virtual Hosts 
12.34.56.789    example.com

this means your IP address corresponds with your domain and only your domain.

I hope I have answered your questions please let me know if I can be of assistance with any more.

this is a great tutorial on setting this up correctly if you are using a VPS that is non managed again call your hosting company if it is a managed server.

https://www.digitalocean.com/community/tutorials/how-to-set-up-apache-virtual-hosts-on-ubuntu-14-04-lts

Sincerely,

Thomas

[mdmoz]

Thank you Thomas for your time and answers.

I added the following code to my htaccess file, but then could not access my site:

Options +FollowSymLinks
RewriteEngine On
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*index\.html?\ HTTP/
RewriteRule ^(.*)index\.html?$ http://www.mydomain.com/$1 [R=301,L]

This is the error when I checked my site:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@********** and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.

checking with my hosting company about the IP error in the meantime.

[BlueprintMarketing]

Ok are you on a CMS what type of code?

removing the code and placing this in RewriteEngine On RewriteRule ^index.html$ / [R=301,L]

http://www.webconfs.com/how-to-redirect-a-webpage.php

http://moz.com/blog/apache-redirect-an-index-file-to-your-domain-without-looping

http://moz.com/learn/seo/redirection

We we have to add the  removed the /

http://stackoverflow.com/questions/13235180/how-to-remove-trailing-slash-for-folder-index-html-when-index-html-is-stri

See this as well

http://dense13.com/blog/2012/10/29/removing-index-html-with-mod_rewrite-in-htaccess/

RewriteEngine On 
RewriteCond %{HTTP_HOST} ^www.mydomain.com$ [NC] 
RewriteRule ^(.*)$ http://mydomain.com/$1 [R=301,L]

Redirect 301 /index.htm http://mydomain.com

[mdmoz]

removed and added RewriteEngine On RewriteRule ^index.html$ / [R=301,L]

now index.html is redirected to / successfully.

How can I ask Mozpro to check again? I can still see duplicate content warning in my Mozpro account.

Should I go for Crawl test menu?

[mdmoz]

Just checked my site at http://www.siteliner.com/

Still see duplicated warning for www.mysite.com and www.mysite.com/index.html

Headache :d

[BlueprintMarketing]

The

duplicated warning for www.mysite.com and www.mysite.com/index.html

will stay for a week

Or unfortunately you would have to delete your campaign or start a new campaign that would let you because moz indexes and checks for those errors once week it would need a new campaign but I would not worry about the errors that showing you now that you have corrected this

Awesome excited about redirect work. I did not know what to tell you if that did not. Work

Okay you can crawl your domain and get very similar results almost the same thing. Using this

https://moz.com/researchtools/crawl-test

If your site is under 500 pages and you want quick confirmation in addition you can use this tool for free up to 500 pages

http://www.screamingfrog.co.uk/seo-spider/

So use this tool

If you want I can verify it as well for you

https://moz.com/researchtools/crawl-test

If it is your first time using the Moz campaign tool and you have no history to worry about you can simply start a new campaign. In addition to these tests

I hope this helps,

Tom

[BlueprintMarketing]

Nice!

Yes this will check it 2 times a day if needed

[BlueprintMarketing]

If you are still having problems could you send me your domain either in the form here or privately?

I am psyched that the redirect worked that is really good.

These other issues I can troubleshoot for you very quickly using some tools I have in addition to the excellent tools Moz has.

Download a free copy of screaming frog just a have it is honestly worth it as it is free and awesome.

Can I be of more assistance,

Tom

[BlueprintMarketing]

It could be cache

Try

http://codeblow.com/questions/getting-rid-of-index-html-from-url-and-adding-www-with-a/

try RewriteEngine On 
RewriteCond %{HTTP_HOST} ^mydomain.com$ [NC] 
RewriteRule ^(.*)$ http://www.mydomain.com/$1 [R=301,L]

Redirect 301 /index.htm http://www.mydomain.com

[eugenecomputergeeks]

I was just about to ask the same question and about to do a 301 redirect myself.  Mine question is should I use www. or just http://site.com?  I know its not good to have both as it could be cited as duplicate content, but what do you all think?   My site is ***techhelp.fake     (not really .fake and its not a TLD or whatever.

[BlueprintMarketing]

Hi

here is how

http://www.webconfs.com/how-to-redirect-a-webpage.php

http://moz.com/learn/seo/redirection

To determine rather not you should redirect to www. or no-www

use http://moz.com/researchtools/ose

to figure out which has more back links pointing to it then 301 redirect to that.

In terms of if one is better than the other there is really not a difference however if they are both equal meeting having no back links or identical.

You might want to go with www. simply because having the subdomain allows you to do some more things like add Google page speed other miscellaneous things in the future.

But really go with the one with the most links if there are no links then choose the one you want going to Google webmaster tools to make sure you add both

examples.com

www.example.com

to Google webmaster tools than pick the same URL you chose to redirect to as your main URL Google will then only send Google bot out looking for that URL

Here

https://support.google.com/webmasters/answer/44231?hl=en

Hope this helps,

Tom