Duplicate Content From Indexing of non- File Extension Page

[TakeshiYoung]

Add a canonical tag to your header so that Google/Bing knows which version of your page they should be indexing.

You can also try looking into where the link to the non-html page is coming from. If it's an internal link, just change it so that Google doesn't continue to crawl it.

[WebbyNabler]

I tried the canonical and when I enter the url without the .html, it doesn't resolve to the url with the .html extension.  I tried an .htaccess reirect...I am stumped, I can't get it to redirect automatically the the .html  I accidentally manually submitted the url to google and manually in submitted it to index and that when this issue began....

[TakeshiYoung]

A canonical tag won't physically redirect you when you visit the page, it just lets the search engines know which is the right page to index.

If you want to actually redirect using .htaccess, try using this code

RewriteEngine On

RewriteCond %{REQUEST_URI} ! .html$

RewriteCond %{REQUEST_URI} ! /$

RewriteRule ^(.*)$ $1.html

[NakulGoyal]

Is this a wordpress based site ? What CMS are you using ? How were you able to get domain.com/sample and domain.com/sample.html be the same page ? Either way, canonical tag is the correct solution in this case. There's no need for a 301 and if you do 301 redirects, you are not really fixing the issue caused by your CMS System.

I would therefore strongly advise to use the canonical tag. That's the intended use of that tag.

[WebbyNabler]

Thank you very for your input!  When I implement into my .htacces what you suggested I get a "Internet 500 Server Error" ?   Maybe it would help if I list what I currently have in my .htaccess  I had to redirect some old domains and did canonical redirects and default non .index....I hope this help, I am at my wit's end...  I also attached a screenshot of the webmaster warning... THANKS!!!

Options +FollowSymlinks
RewriteEngine On

RewriteCond %{HTTP_HOST} ^hanneganconstructionllc.com [NC]
RewriteRule ^(.*)$ http://hanneganremodeling.com/$1 [L,R=301]

RewriteCond %{HTTP_HOST} ^www.hanneganconstructionllc.com [NC]
RewriteRule ^(.*)$ http://hanneganremodeling.com/$1 [L,R=301]

RewriteCond %{HTTP_HOST} ^hremodeling.com [NC]
RewriteRule ^(.*)$ http://hanneganremodeling.com/$1 [L,R=301]

RewriteCond %{HTTP_HOST} ^www.hremodeling.com [NC]
RewriteRule ^(.*)$ http://hanneganremodeling.com/$1 [L,R=301]

RewriteEngine on
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /index.html\ HTTP/
RewriteRule ^index.html$ http://www.hanneganremodeling.com/ [R=301,L]

RewriteEngine On
Options +FollowSymlinks
RewriteBase /
RewriteCond %{HTTP_HOST} ^hanneganremodeling.com$ [NC]
RewriteRule ^(.*)$ http://www.hanneganremodeling.com/$1 [R=301,L]

Options +FollowSymLinks
RewriteEngine On
RewriteBase /

u1YvXLR

[WebbyNabler]

No it's not a wordpress, it was created with Dreamweaver.  I didn't make sample and sample.html same page, but google is treating it that way....  I have implemented the 301, so I guess I just have to wait for a crawl

[Mike.Goracke]

"I accidentally manually submitted the url to google and manually in submitted it to index and that when this issue began...."

It sounds like you accidently added this URL to the index. You can follow the procedure outlined below to request Google remove the specific URL from the index:

https://support.google.com/webmasters/bin/answer.py?hl=en&answer=59819

I checked your site's structure using Screaming Frog and it does not appear that you are linking to any non-.html versions. If I perform a scan using one of your non-.html pages, it appears that it only links to itself.

Since you have the 301 redirect in place, you can choose to wait it out and Google should correct things eventually; otherwise, requesting Google remove the URL is a faster... PERMANENT process.

Good luck.

Mike

[WebbyNabler]

Thanks Mike, you are awesome!  I actually was thinking to do that, but I was concerned that it might have some larger implications?

I also just resubmitted a sitemap so hopefully that "might" speed up the crawl process...

Thanks again!

[TakeshiYoung]

You repeat this code a few times, maybe that's the problem? Pretty sure you only need it once:

RewriteEngine On
Options +FollowSymlinks
RewriteBase /

The line:

RewriteEngine On

Also only needs to be included once in an htaccess file. You may want to remove all the other instances.

Try adding this code at the very top, after the first "RewriteEngine On":

RewriteCond %{REQUEST_URI} ! .html$
RewriteCond %{REQUEST_URI} ! /$
RewriteRule ^(.*)$ $1.html

[WebbyNabler]

I get a server error when I do this?  Sooo confused...  Here is the htaccess changes I made.  FYI...I have removed the code you told me to put in there temporarily so the site's not down.  I attached the server error screenshot too...

Options +FollowSymlinks
RewriteEngine On

RewriteCond %{REQUEST_URI} ! .html$
RewriteCond %{REQUEST_URI} ! /$
RewriteRule ^(.*)$ $1.html

RewriteCond %{HTTP_HOST} ^hanneganconstructionllc.com [NC]
RewriteRule ^(.*)$ http://hanneganremodeling.com/$1 [L,R=301]

RewriteCond %{HTTP_HOST} ^www.hanneganconstructionllc.com [NC]
RewriteRule ^(.*)$ http://hanneganremodeling.com/$1 [L,R=301]

RewriteCond %{HTTP_HOST} ^hremodeling.com [NC]
RewriteRule ^(.*)$ http://hanneganremodeling.com/$1 [L,R=301]

RewriteCond %{HTTP_HOST} ^www.hremodeling.com [NC]
RewriteRule ^(.*)$ http://hanneganremodeling.com/$1 [L,R=301]

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /index.html\ HTTP/
RewriteRule ^index.html$ http://www.hanneganremodeling.com/ [R=301,L]

RewriteBase /
RewriteCond %{HTTP_HOST} ^hanneganremodeling.com$ [NC]
RewriteRule ^(.*)$ http://www.hanneganremodeling.com/$1 [R=301,L]

m9Kq8yJ

[Mike.Goracke]

Maybe give this a whirl:

If URL does not contain a period or end with a slash

RewriteCond %{REQUEST_URI} !(.|/$)

append .html to requested URL

RewriteRule (.*) /$1.html [L]

[WebbyNabler]

Nope.  Didn't work.  I am seriously about to lose my mind with this....

[WebbyNabler]

Oh, and my site auditor is seeing it as a directory with a file in it???  Ugghhh....

[TakeshiYoung]

Try my code without all the other redirects, and see if it works. If it does, then add back the other redirects one by one until everything works.

[WebbyNabler]

That caused the same "500 Internal Server Error" .......  

[TakeshiYoung]

Try:

RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^([^.]*[^./])$ /$1.html [R=301,L]

[Mike.Goracke]

If that does not work, give this a whirl:

RewriteCond %{REQUEST_URI} !\.[a-zA-Z0-9]{3,4}

RewriteCond %{REQUEST_URI} !/$

RewriteRule ^(.*)$ $1.html

[WebbyNabler]

Nope.

[WebbyNabler]

Nope. .....good lord....

[Mike.Goracke]

LAME! You may just want to let the 301 redirect you have in place take its course or remove the URL from Google's index since it was added by mistake anyway.

Mike