Does this site have a duplicate content issue?

[rja214]

Yes, the page http://www.customgia.com/fashion-beaded-jewelry/shop-for/beaded-bracelets/classic-bracelet/AADE17E9BB964352B7A3912294BB5DF8 is a unique product page with a unique description.  The designer page you referenced: http://www.customgia.com/design-your-own-jewelry/classic-bracelet/AADE17E9BB964352B7A3912294BB5DF8 is also unique because it loads the jewelry design in the product page.

On every product page the "Design It!" button opens a flash-based designer page that let's the user edit that particular design.  Unfortunately, the Moz crawler (and I assume Google) considers these pages duplicates of http://www.customgia.com/design-your-own-jewelry/classic-bracelet/ (or one of the other five jewelry patterns).  The fact that every designer page loads a unique jewelry design does not seem to matter.  The best (and most costly) solution, I suppose would be to change all the flash code to html but that isn't happening anytime soon.

The

tag on all the designer pages is either "Classic Bracelet", "Classic Necklace", "Simple Necklace", "Simple Bracelet", "Pendant Necklace" or "Drop Earrings" (depending on which pattern was used to create the design).  Maybe changing the

to something like:  Redesigning "Garnet Pearl Bracelet with Silver Hearts" would help tell Google these pages indeed differ from one another - but I think the content below the

will still be considered duplicate.  If I use canonical tags on these pages, is there any point in creating dynamic

tags if it doesn't improve the user experience?  Thanks again!

[MickEdwards]

Even if the 'design' part was not flash the textual content is pretty much identical.  There is no benefit for it to be indexed so canonical to the main directory URL would make sense.  Then add some good text to those main pages.

Personally I would only use H1 for user experience rather than keywordy as they don't carry much weight.

[Dr-Pete]

Each of these problems may have a unique solution, so it gets complicated. Regarding the "design your own" pages, I'm seeing over 5K of those URLs in the search index, and they do probably look very similar. Since these are not the core product pages, I'd strongly consider using META NOINDEX on them. I find that Robots.txt does not do a good job of blocking content that has already been indexed, in most cases. You can add the meta tag dynamically in your code, hopefully, so that just a few lines of code will serve all of these pages.

While these pages aren't "true" duplicates, they look similar enough that, at the scale of your site, they really are diluting your ability to rank. In extreme cases, if you're also serving up product variations, paginated search results, etc., you could even run into Panda issues. Whether or not this is your core problem, from an SEO perspective, cleaning it up can't hurt, and may make it easier to find other problems.

[Everett]

I agree with Dr. Pete here, though I think the easiest solution would be to simply block the entire /design-your-own-jewelry/* directory from being indexed using robots.txt and, to Dr. Pete's point, you'll want to remove that directory from the index in both Bing and Google webmaster tools, as discussed here:

http://googlewebmastercentral.blogspot.com/2007/04/requesting-removal-of-content-from-our.html (see the section under "entire directory")

Something I think about with regard to robots.txt block Vs meta robots block is crawl budget. Google has to access a page to see the meta noindex tag, while a single disallow statement in the robots.txt file can save Googlebot the hassle of visiting potentially thousands of unnecessary pages.

If down the road you figure out a way to put custom content on those pages and want to try and rank for things like "Custom Garnet Pearl Bracelet" or "Design Your Own Beaded Bracelet" then I'd look into some of the other options discussed here. Until then I feel they would just be complicating something as simple as the need to remove very thin, mostly duplicate content from the index.

[Dr-Pete]

I've had a lot of issues where, if pages were already indexed, Robots.txt did a poor job of removing them. Absolutely agree on the crawl budget issue and it's a whole lot easier to remove a folder in Robots.txt, but I've just had a bunch of odd problems with Robots.txt at large scale. If I actually had to do it on my own site, I'd probably:

(1) META NOINDEX the pages

(2) Monitor removal

(3) Once removal was progressing well (80%+), then add to Robots.txt

[Everett]

That sounds like a winning plan Dr. Pete, though I'd append 2.1 "Request removal of directory in Bing and Google webmaster tools".

[rja214]

Thank you for the excellent advice.  We put the META NOINDEX tags into place this morning.  The URL removal request is next but I have a slight change to what Everett outlined above.

The six designer pages that are accessible from the homepage: /design-your-own-jewelry/classic-necklace, /design-your-own-jewelry/simple-necklace, /design-your-own-jewelry/classic-bracelet, /design-your-own-jewelry/simple-bracelet, /design-your-own-jewelry/pendant-necklace, /design-your-own-jewelry/drop-earrings are not duplicates and the Moz crawl did not show them as duplicates.  All the other designer pages are considered duplicates of each other or duplicates of one of these six pages.  So we put INDEX, NOFOLLOW on these six pages to keep them indexed. I think the removal request should follow suit.

What's your opinion on placing a removal request for each of the following? -  /design-your-own-jewelry/classic-necklace/, /design-your-own-jewelry/simple-necklace/, /design-your-own-jewelry/classic-bracelet/, /design-your-own-jewelry/simple-bracelet/, /design-your-own-jewelry/pendant-necklace/, /design-your-own-jewelry/drop-earrings/.  Correct me if I'm wrong, but this should remove all the designer pages from the index except the six that are accessible from the homepage, giving those six pages a chance to rank.

[Dr-Pete]

Oh, sorry - yeah, this is why these questions can be dangerous in the scope of Q&A. If some of the pages in that virtual folder are main nav pages/links, then it's definitely going to look weird to block them (it's a mixed signal, at best). I'm not sure I fully understand the site structure, but my gut reaction is to leave those indexed. The wild-cards should work - the other option would be to give them each their own shorter URLs and not put them in the "/design-your-own-jewelry" folder, but that can be a ton of work, depending on how your site is built (plus, you'd have to 301-redirect the old URLs, which opens up a whole new mess).

[rja214]

Okay, last question on this (I hope).  As far as I can tell, Google's URL removal tool does not support the use of wildcards.  And according to their removal requirements, I can't remove an entire directory unless that directory is already blocked in the robots.txt.  So before I submit the removal request for: http://www.customgia.com/design-your-own-jewelry/classic-necklace/, I have to add:  Disallow: /design-your-own-jewelry/classic-necklace/ to robots.txt.  Is this the right way to do this?  And again, thanks for helping me with this.

[Dr-Pete]

This is where things get a bit dicey - I'm not 100% sure that won't remove the main page, too (and how Google handles the trailing "/"). You might need a "/*" wild-card in the Robots.txt. Frankly, I'd ease into it with just one directory. These things never seem to work quite the way in practice that we all say they should in theory.