Welcome to the Q&A Forum

If you block the pages from being crawled, you are also telling the search engines to not index the pages (they don't want to include something they haven't looked at). So yes, the traffic numbers from organic search will change if you block the pages in robots.txt.

Hi Monique, well done, you already know the answer Use a Canonical Tag for all the duplicate pages, to point back to your main login page http://www.quicklearn.com/maven/login.aspx (adding a couple of unique sentences may not make the pages unique enough to not be classed as duplicate, as duplicate includes very similar pages). Regards, Simon ~ (PS. sorry for the lack of paragraphing here, doesn't work on an iPad)

Hi Yannick, I realize that you can't solve duplicate content issues with a sitemap, but an XML sitemap does (according to my understanding) tell Google how to organize the site's pages -- and in the case of a site that has too many URLs on it for Google to crawl them all, I imagine it would also influence the priority assigned to the indexing of a given page (meaning that a URL listed in the sitemap would likely not be skipped over by Google, but some random, less important search results page not on the sitemap may be overlooked). Unfortunately I'm not sure that we can so easily shrug off the whole dupe content issue, since we are talking dozens of URLs per page of content. So, so many. There's no way Google can handle all of it, and our SEO link juice has got to be so diluted as to be useless. So we'd really like to fix this if at all possible. Do you have any advice re: whether to go one direction (XML sitemap with URLs that do not match our "vanity" desired canonicals) or another (rel="canonical" for our pretty vanity SEO'd URLs)? Or both? We've got neither at the moment. Thanks for your help.

Perfect! Glad to hear it's all fixed

HI Dillon, I'm so sorry that I came across as harsh and rude. That certainly was not my intention, and I do apologize. In retrospect, I can see how that could have happened, and will work to make sure it doesn't happen again. As Ryan and Robert have mentioned, I am quite active in Q&A and usually leave a much better impression than this. I'm sorry you received this impression, especially as a brand new user. Keri Morgret

So does anyone of You have some opinion or maybe a right answer/solution for this? Thanks!

Hi Daniel, Whenever I hear "Nothing is redirecting when I add this to the .htaccess file" I have one thought: Is the site using Joomla, Drupal or some other CMS that will overwrite .htaccess rules externally? Sha

Scrapebox is an excellent tool for blog and forum discovery. SENukeX doesn't really help at all in that department and is really only a decent tool if you find creative ways to use it like building your own blog networks.

Hi Keri...thanks for sharing some insight to my response as well ;>) Is due to the Panda Update why they should be indexed -- since the panda update I believe now wants to see the contact page and those 2 other pages I think ...hummm thanks for clarity !

It only works for Google news, it is not for organic, and is not for other search engines. it works different as i understad it. if the daily news has a story taged as original souce, then it would tell the SE' that it belongs to their organization, not any one page. Get the difference here. they may have the story on breaking news, later on front page, then later still on buiness news, and still later in archives. so canonical would not work well for them.

subdomain or different domain it's still an identical site. it would be better to have it on a co.uk domain, i would convince the client to do that - why do they want it on a subdomain? Anyway, subdomain, main domain in a directory, or different domain. Google will be fine with it as long as one site is geo targeted for the US and the other site is geo targeted for the UK. Duplicate content is filtered not penalized, so it is getting properly filtered this way. One site showing in the US and one showing in the UK.

excellent points. thanks very much for the inputs

I checked a few sites myself and see your point. Good argument, Ryan! This is something  I can improve.

I just did this for my site (strikemodels.com). We bought bderc.com and did a 301 redirect over to the appropriate pages of Strike Models. If I go into OSE and look for only links with a 301, I see a bunch of links from bderc.com. I need to put bderc.com into OSE and to see who is linking to bderc instead of me. So if you do this for onemainfinancial, you're going to see a ton of 301 from citibank -- not sure if you were wanting that, or those who were linking to citibank before it was 301'd.

It was only recent that Google ancouced that they now allow it.

What do you mean by over-optimized?

If the other site is outranking yours, it would mean that for whatever reason, Google has decided it is better quality, but not necessarily determined one or the other to be a scraper. That could be based on any combination of the hundreds of factors Google uses to determine position in search results.  It may just be that it has more/better inbound links. If you focus on keeping only quality original content on your site, getting good links and mall of your on-page SEO is in good shape then you shouldn't have to worry about scrapers. Google will find and devalue the duplicates eventually. If this is one of those cases where yours is clearly the original and the scraper is outranking yours for whatever reason, you may want to consider filing a DMCA report with the site's hosting company.  You can usually find that info with a whois search. You can also try submitting a spam report to Google here https://www.google.com/webmasters/tools/spamreport