Noindexing Duplicate (non-unique) Content

[Philip-DiPatrizio]

I'm not 100% sure Google will understand you if you leave off the slashes.  I've always added them and have never had a problem, so you want to to type: /oahu/waianae-makaha-condos/

Typing that would NOT include the neighborhood URL, in your example.  It will only remove everything that exists in the /waianae-makaha-condos/ folder (including that main category page itself).

edit >> To remove the neighborhood URL and everything in that folder as well, type /oahu/waianae-makaha/maili-condos/ and select the option for "directory".

edit #2 >> I just want to add that you should be very careful with this.  You don't want to use the directory option unless you're 100% sure there's nothing in that directory that you want to stay indexed.

[khi5]

thx, Philip. So you are saying if I use the directory option that will ensure the paginated pages will also be taken out of the index like this page: /oahu/waianae-makaha/maili-condos/page-2

[khi5]

so I will remove directory for "/oahu/waianae-makaha/maili-condos/" and that will ensure removal of "/oahu/waianae-makaha/maili-condos/page-2" as well?

[Philip-DiPatrizio]

Yep.  Just last week I had an entire website deindexed (on purpose, it's a staging website) by entering just / into the box and selecting directory.  By the next morning the entire website was gone from the index

It works for folders/directories too.  I've used it many times.

[khi5]

removing directory "/oahu/waianae-makaha-condos/" will NOT remove "/oahu/waianae-makaha/maili-condos/" because the silo "waianae-makaha" and "waianae-makaha-condos" are different.

HOWEVER,
removing directory " /oahu/waianae-makaha/maili-condos/" will remove "/oahu/waianae-makaha/maili-condos/page-2" because they share this silo "waianae-makaha"

Is that correctly understood?

[khi5]

for some MLS result pages I have a BUNCH of pages and I want to remove from index with 1 click as opposed to having to include each paginated page. Example: http://www.honoluluhi5.com/oahu/honolulu/metro/waikiki-condos/page-52 I simply include"/oahu/honolulu/metro/waikiki-condos/" and that will ALSO remove from index this page: http://www.honoluluhi5.com/oahu/honolulu/metro/waikiki-condos/page-52 - is that correct?

[Philip-DiPatrizio]

Yep, you got it.

You can think of it exactly like Windows folders, if that helps you stay focused.  If you have C:\Website\folder1 and C:\Website\folder12.  "noindexing" \folder1\ would leave \folder12\ alone because they're not in the same directory.

[Philip-DiPatrizio]

Yes.  It will remove /page-52 and EVERYTHING that exists in /oahu/honolulu/metro/waikiki-condos/.  It will also remove everything that exists in /page-52/ (if anything).  It trickles down as far as the folders in that directory will go.

**Go to Google search and type this in: **site:honoluluhi5.com/oahu/honolulu/metro/waikiki-condos/

That will show you everything that's going to be removed from the index.

[khi5]

thx, Philip. Most helpful. I will get on it

[khi5]

based on Google's own guidelines it appears to be a bad idea to use the removal tool under normal circumstances (which I believe my site falls under): https://support.google.com/webmasters/answer/1269119

It starts with: "The URL removal tool is intended for pages that urgently need to be removed—for example, if they contain confidential data that was accidentally exposed. Using the tool for other purposes may cause problems for your site."

[Philip-DiPatrizio]

Good find.  I've never seen this part of the help section.  Their resonating reason behind all of the examples seems to be "You don’t need to manually remove URLs; they will drop out naturally over time."

I have never had an issue, nor have I ever heard of anyone having an issue, removing URLs with the Removal Tool.  I guess if you don't feel safe doing it, you can wait for Google's crawler to catch up, although it could take over a month.  If you're comfortable waiting it out, have no reasons to rush it, AND feel like playing it super safe... you can disregard everything I've said

We all learn something new every day!

[khi5]

lol - good answer Philip. I hear you. What makes it difficult is the lack of crystal clear guidelines from search engines....it is almost like they don't know themselves and each case is sort of on a "what feels right" basis.....

[AlanMosley]

Remember that if you no-index pages, any link you have on your site pointing to those pages is wasting its link juice.

This looks like a job for Canonical tag

[khi5]

Hi Alan, thx for your comment. Let me give you an example and if you have a though that's be great:

1. Condos on Island: http://www.honoluluhi5.com/oahu-condos/
2. Condos in City: http://www.honoluluhi5.com/oahu/honolulu-condos/
3. Condos in Region: http://www.honoluluhi5.com/oahu/honolulu/metro-condos/

Properties on the result page for 3) are all in 2) and all properties within 2) is within 1). Furthermore, for each of those URL, the paginated pages (2 to n) are all different, since each property is different, so using canonical tags would not be accurate. 1 + 2 + 3 are all important keywords.

Here is what I am planning: add some unique content to the first page in the series for each of those URL and include just the 1st page in the serious to the index, but pages 2 to n I will keep "noindex, follow" on. Argument could be "your MLS result pages will look too thin and not rank" but the other way of looking at it is "with potentially 500 or more properties on each URL, a bit of stats on page 1 will not offset all the MLS duplicate data, so even though the page may look thin, only indexing page 1 is best way forward".

[Philip-DiPatrizio]

Sounds like you should actually be using rel=next and rel=prev.

More info here: http://googlewebmastercentral.blogspot.com/2011/09/pagination-with-relnext-and-relprev.html

[khi5]

Thx ,Philip. I am using already, but I thought adding "noindex, follow" to those paginated pages (on top of rel=next prev") will increase likelihood G will NOT see all those MLS result pages as a bunch of duplicate content. Page 1 may look thin, but with some statistical data I will soon include it is unique and that uniqueness may offset lack of indexed MLS result pages.....not sure if my reasoning is sound. Would be happy to hear if you feel differently

[AlanMosley]

If you no index, I don't think Next Previous will have any affect.

If they are different then and if the keywords are all important why no-index?

[khi5]

http://www.honoluluhi5.com/oahu-condos/  - this is an "MLS result page". That URL will soon have some statistics and it will be unique (I will include in index). All the paginated pages (2 to n) hardly has any unique content. It is great layout, users love it (ADWords campaign average user spends 9min and views 16 pages on site), but since it is MLS listings (shared amongst thousands of Realtors) Google will see "ah, these are duplicate pages, nothing unique". That is why I plan to index page 1 (the URL I list) but all paginated pages like: http://www.honoluluhi5.com/oahu-condos/page-2) I will keep as "noindex, follow". Also, I want to rank for this URL: http://www.honoluluhi5.com/oahu/honolulu-condos/ which is a sub-category of the first URL and 100% of the content is exactly the same as the 1st URL. So, I will focus on indexing just the 1st page and not the paginated pages. Unfortunately, G cannot see value in layout and design and I can see how keeping all pages indexed could hurt my site.

Would be happy to hear your thoughts on this. I launched site 4 months ago, more unique and quality content than 99% of other firms I am up against, yet nothing happens ranking wise yet. I suspect all these MLS pages is the issue. Time will show!

[AlanMosley]

There is nothing wrong with having duplicate content. It becomes a problem when you have a site that is all or almost all duplicate or thin content.

Having a page that is on every other competitors site will not harm you, you just may not rank for it.

but no indexing can cause lose of link juice as all links pointing to non indexed pages waste there link juice. Using noindex,follow will return most of this, but still there in no need to no-index

[khi5]

http://moz.com/blog/handling-duplicate-content-across-large-numbers-of-urls  - that is Rand's whiteboard Friday a few weeks ago and I quote from the transcripts:

"So what happens, basically, is you get a page like this. I'm at BMO's Travel Gadgets. It's a great website where I can pick up all sorts of travel supplies and gear. The BMO camera 9000 is an interesting one because the camera's manufacturer requires that all websites which display the camera contain a lot of the same information. They want the manufacturer's description. They have specific photographs that they'd like you to use of the product. They might even have user reviews that come with those.

Because of this, a lot of the folks, a lot of the e-commerce sites who post this content find that they're getting trapped in duplicate content filters. Google is not identifying their content as being particularly unique. So they're sort of getting relegated to the back of the index, not ranking particularly well. They may even experience problems like Google Panda, which identifies a lot of this content and says, "Gosh, we've seen this all over the web and thousands of their pages, because they have thousands of products, are all exactly the same as thousands of other websites' other products."