I try to apply best duplicate content practices, but my rankings drop!

[LukeyJamo]

Hey,

An audit of a client's site revealed that due to their shopping cart, all their product pages were being duplicated.

http://www.domain.com.au/digital-inverter-generator-3300w/

and

http://www.domain.com.au/shop/digital-inverter-generator-3300w/

The easiest solution was to just block all /shop/ pages in Google Webmaster Tools (redirects were not an easy option).

This was about 3 months ago, and in months 1 and 2 we undertook some great marketing (soft social book marking, updating the page content, flickr profiles with product images, product manuals onto slideshare etc).

Rankings went up and so did traffic.

In month 3, the changes in robots.txt finally hit and rankings decreased quite steadily over the last 3 weeks.

Im so tempted to take off the robots restriction on the duplicate content.... I know I shouldnt but, it was working so well without it?

Ideas, suggestions?

[AlanMosley]

I have been telling other not to do what you have done. what is better is you use "no-index follow" tags instead.

Link juice flows into your pages blocked by robots though links to them never to be seen again. If you use the noindex-follow meta tag you allow the link juice to flow in and out.

The best idea is not to have the duplicates, after thet you should use a canonical tag, if that is not posible then use the noindex, follow tag.

[LukeyJamo]

The shopping cart WP plugin is creates the /shop/product pages automatically. I have very little control over them.

Instead the developer has created post pages and inserted the product short codes (this gives the /product effect). I have far more control over these pages, and as such they are far better for SEO.

Do you know of a way I can no index/follow, all /shop pages in the robots.txt?

[AlanMosley]

I work with Microsoft Technolgies, I dont work with CMS like WP.

On IIS i would use a outgoing url rewrite url to insert the meta tag. You can do this without touching the website.

Are you by any luck hosting on IIS or are you on Apache?

[LukeyJamo]

No it's Apache.

Would you guys say it's best to just 301 rewrite all /shop/product to /product? Then unblock from robots.txt?

[AlanMosley]

I dont understand the difference, is this the reason behind the duplicates?

[LukeyJamo]

The difference between blocking something in robots.txt and a 301?

The duplicates are actively created.

When the products are added to the cart plugin, they automatically create the /shop/product page. These pages were horrible for SEO, and as they were automatically created they could not be edited easily (the plugin developers clearly had no SEO understanding).

My client's developer created a new WP Post for every product and added in a shortcode calling the product from the plugin. This created the duplicate. As this is a wordpress post, the SEO was far more adaptable.

[AlanMosley]

Yes i would 301 then

Are there any links pointing to the shop/ version, internal links? then i would fix them also, as 301's leak link juice you should create link that go directly to the destionation page where you can.

[LukeyJamo]

No, it was a very new site. There was only 10 links all to the root.

Thanks for your help.

[AlanMosley]

No I mean internal links,

if you do a 301, your internal links poiting to shop/ urls will still work, but they will lose a little link juice when they are 301 redirected. you should point them to the final destination the non shop url.

[Dr-Pete]

Agreed with Alan (deeper in the comments) - you may have cut off links to these pages or internal link-juice flow. It would be much better to either 301-redirect the "/shop" pages or use the canonical tag on those pages. In Apache, the 301 is going to be a lot easier - if "/shop/product" always goes to "/product" you can set up a rewrite rule in .htaccess and you don't even need to modify the site code (which site-wide canonical tags would require).

The minor loss from the 301s should be much less than the problems that may have been created with Robots.txt. As Alan said, definitely re-point your internal links to the canonical (non-shop) version.