Duplicate content on URL trailing slash

[yacpro13]

Hello,

Some time ago, we accidentally made changes to our site which modified the way urls in links are generated. At once, trailing slashes were added to many urls (only in links).

Links that used to send to
example.com/webpage.html

Were now linking to
example.com/webpage.html/

Urls in the xml sitemap remained unchanged (no trailing slash).

We started noticing duplicate content (because our site renders the same page with or without the trailing shash). We corrected the problematic php url function so that now, all links on the site link to a url without trailing slash.

However, Google had time to index these pages. Is implementing 301 redirects required in this case?

[BlueprintMarketing]

I have provided the Apache and Nginx configurations you would need in addition to a URL that will convert

Apache Htaccess to Nginx

The instructions are right here

Remove Trailing Slash

Just like with the WWW example, some prefer to remove the trailing slash. It's a commonly debated question that you'll find around the Internet, but it just depends on what you prefer.

Remember, though, your browser and even your server, by default, add a trailing slash to a directory. It is done for a reason. If you must strip the trailing slash, though, this is how you would do it:

<code class="hljs apache">RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_URI} !(.*)$
RewriteRule ^(.*)$ http://www.domain.com/$1 [R=301,L]</code>

For Nginx

nginx configuration location ~ (.)$ { } location / { if (!-e $request_filename){ rewrite ^(.)$ http://www.domain.com/$1 redirect; } }

The explanation for this rule is the same as it is for when we want to add a trailing slash, just in reverse. We can also specify specific directories that we don't want apply this rule over.

<code class="hljs apache">RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_URI} !directory/(.*)$
RewriteCond %{REQUEST_URI} !(.*)$
RewriteRule ^(.*)$ http://www.domain.com/$1 [R=301,L]</code>

For Nginx

nginx configuration location ~ directory/(.)$ { } location ~ (.)$ { } location / { if (!-e $request_filename){ rewrite ^(.*)$ http://www.domain.com/$1 redirect; } }

Please see the note about mod_dir and the DirectorySlash directive in the previous example. You might need to turn this directive off.

HTaccess converter for Apache to Nginx configuration.

http://winginx.com/en/htaccess

https://www.maxcdn.com/one/tutorial/remove-trailing-slash/

https://www.crucialhosting.com/knowledgebase/htaccess-apache-rewrites-examples

https://moz.com/community/q/how-to-remove-trailing-slashes-in-urls-using-htaccess-apache

https://gist.github.com/nathos/4645164

[GlobeRunner]

If you have changed the URLs with trailing slashes, then there are a few things you'll want to do:

• make sure all the internal links on your site are updated to point to the proper version.

• make sure that the sitemap.xml file(s) are correct, pointing to the proper version.

• set up 301 permanent redirects so that the ones with the slash are redirecting to the old URLs.

As long as you have corrected the links internally, updated the sitemap file, and set up the 301 redirects, everything should go "back to normal" within a fairly short period of time. You will need to give it time, though, as Google will need to re-crawl all of those URLs and get it all ironed out.

[MattRoney]

Hi yacpro13! Did Eric or Thomas answer your question, and if so, would you mind marking one or both responses as a "Good Answer?"

Otherwise, what questions do you still have?

[yacpro13]

Hi Eric,

I was at Step 3 of your 3 Step plan, looking for confirmation as to whether or not the 301 redirects were required in this situation.

Thanks!

[BlueprintMarketing]

Yes you want to have it match the canonical tag so most effective method is to 301 redirect so they match the canonical tag site map and robots.txt etc.  You can use a Regex code like this at the end of the URL /?$ in the case of category URLs it will allow them when needed.

if you use the proper 301 you will not have to deal with the category issue anyway.

rel="canonical" href="https://moz.com/community/q/duplicate-content-on-url-trailing-slash" />

I hope this is able to shed more light on the issue and great answer Eric.

Hope I was of help,

Tom