Welcome to the Q&A Forum

Hi Mark,

Your concerns are very much true and unfortunately there is no absolute solution for this issue.However, the following might help you face and handle the issue:

1. Use Google authorship.

This will help the search engines recognize the correct or the rightful owner of the content. Here you go for more:

http://www.google.com/insidesearch/features/authorship/index.html

2. Have the copyright notices published along with the content. Give clear syndication guidelines like, an attribution in the form of a link pointing to your webpage (that the content resides on) if someone wants to publish on their website.

3. You can always file DMCA complaints right from the Google webmaster tools account.

4. You can use Google Alerts to watch out for any of your content being stolen and published elsewhere.

Here you go for more in this regard:

http://blog.kissmetrics.com/find-remove-stolen-content/

Hope the above help.

Best,

Devanur Rafi.

Hi there, though a single Sitemap.xml file can accommodate upto 50K URLs, it is not uncommon to go for multiple Sitemap.xml files for many purposes even with few hundreds on each.

You need to come up with a total of 4 Sitemap files and one among these would be an index sitemap that lists the other 3 Sitemap.xml files with URLs.

Here you go for more: http://www.sitemaps.org/protocol.html

Best,

Devanur Rafi

You are most welcome Mark. At some point or the other, we all (at least people who come up with original content) would have faced the content scraping or stealing issue. Its all the part of the game. Being the rightful owners of the content, we should not leave any stone unturned while trying to protect our rights for our content. Personally, I have seen the copyright notice work well with manual content stealers but, for those who use automated tools to scrape and publish content, there is no definite method to stop them. Wish you all the very best for all your endeavors.

Best,

Devanur Rafi.

Thanks for the response my friend. The problem without an index sitemap file is, when you have to resubmit multiple sitemap.xml files in webmaster tools account, you will have to resubmit each of them at a time. With an index sitemap file, you just need to submit the index file and it would take care of the job.

Here you go for more: https://support.google.com/webmasters/answer/71453?hl=en

Best,

Devanur Rafi

Hi,

I would like to address the following part of your original query without even going to the article that you referred to:

"everyone seems to agree that you shouldn't canonicalize all pages in a series to the first page"

The reason for this is, if you canonicalize all the pages in a series to the first page, you are giving a hint to Google that only the first page is what you are concerned of and it should be indexed. With this, all the non-canonical pages will be taken out from the index which you would not want to happen especially when the content of these pages is unique.

So depending on your requirement you can opt for either a 'view all method' or 'rel=prev/next' method for your pagination requirements.

Good luck.

Best,

Devanur Rafi

Hi there, robots.txt file is one of the initial things that search engine spiders look at when they visit your website and a reference to the Sitemap.xml file in there will aid the search engine spider to quickly access to important URLs on your website then and there.

Best,

Devanur Rafi

Hi, there is no added advantage or disadvantage with www in the URL. You can go either way, with or without www but be consistent throughout the website. You also set your preferred version in Google Webmaster Tools account. Make sure only one version, either www or without www returns an HTTP header status code 200 and not both. If suppose, http://www.abc.com is your preferred version, then only http://www.abc.com should return 200 and not both (http://abc.com). In this case, http://abc.com should be redirected to http://www.abc.com via a server-side 301 permanent redirection. This way, you can take care of your URL canonicalization issues.

One more thing, while doing any link building/earning activities, please make sure to use the preferred version for backlinking purposes. Please feel free to post any queries that you may have in this regard.

Good luck.

Best regards,

Devanur Rafi

No problem my friend. You are most welcome. Yes, you just need to give the location of your sitemap.xml file as given below:

Sitemap: http://example.com/sitemap_location.xml

Here you go for more:
https://support.google.com/webmasters/answer/183669?hl=en

Hi, its time to check your titles and descriptions. With all respect for Yoast plugin, I would still recommend to write the page titles and descriptions manually at least for the most important pages on your website. Google changing the titles and descriptions is not new and all that we can do is try and test different titles and descriptions.

Here you go for more on this..

1. https://yoast.com/google-page-title/

2. http://searchenginewatch.com/article/2342232/Why-Google-Changes-Your-Titles-in-Search-Results

3. http://wordpress.org/support/topic/plugin-wordpress-seo-by-yoast-title-changes-to-individual-pages-categories-tags-posts-is-not-working

and here..

4. https://productforums.google.com/forum/#!topic/webmasters/K6D14caef7U

Hope they help.

Best regards,

Devanur Rafi

If you are talking about the sitemap for the visitors on your website, if you think the newly added pages are going to be helpful to them, you can update your visitors sitemap accordingly. But the Sitemap.xml file is a supplemental indexing tool meant for the search engines to find the pages on your website easily and needs to be updated and resubmitted to search engines using webmaster tools accounts whenever new pages are added to your website.

Hope that helps.

Best,

Devanur Rafi

Yes, of course my friend, Google has to crawl the page to see the page-level meta robots tag but till date I have not seen any page in Google's index that has been blocked using the robots.txt file and page-level meta robots tag. Password protecting your .htaccess file would be an overkill if you just want Google not to index a page. If you want Google to remove any particular page from its index, you can get it done from webmaster tools account. Here you go for more: https://support.google.com/webmasters/answer/1663419?hl=en

Good Luck to you my friend.

Best regards,

Devanur Rafi

Hi Chris,

Even before addressing the duplicate page Title issue, these titles seem to be very long and stuffed with phrases and look spammy.

As you know, page Titles are one of the most important on-page optimization factors that can accommodate the target terms as they  show up in search results. Its better if it is around 55-60 characters in size including spaces with the target term towards the beginning. If the title has to go beyond 55-60 characters, then the essence should be captured in the first 55-60 characters including spaces as Google truncates anything beyond around 55-60 characters and the search engine user  cannot see beyond those ~60 characters in the search results.
Though Google seems to truncate titles in search results based on pixel width(512pixels) and not character limit, in general the 55-50 character limit is a safe bet while writing optimized page titles for your pages.

Coming to the duplicate page Titles issue on your Wordpress website, you can use plugins like Yoast SEO or All-in-one SEO to customize the titles. Here is the deal, no two pages on the website should have same Titles or Descriptions. They should be unique by at least a word or few characters. You can have a look at your Google Webmaster Tools account for duplicate page Titles and Descriptions and act accordingly. And of course, Screamingfrog SEO spider is a superb tool for the purpose. Hope these help.

Best regards,

Devanur Rafi

Hi,

I agree with Dennis here.

One question that you can ask yourself:

"Why should Google have two websites with substantially similar content in their index?"

So, the answer would be to come up with substantially unique content on both the websites so that both of them can pass Google's radar safely  and justify their presence in their index.

In the past, I worked with a client who had multiple websites operating in the same niche but catering to different states in the US. With almost identical content on many of them, they were not at all ranking good in the search engines overall. We had the content rewritten in such a way that all of them had unique content (please note that all these domains still operated in the same niche but with slightly different keywords/phrases based on the differences in naming conventions in different states). This worked wonders. The results exceeded all the expectations of the client. There was a dramatic increase in the total number of pages from these domains in Google's index along with a whopping increase in overall ranking for many of the competitive terms in the search engines.

With substantially unique, updated, highly relevant and useful content, you will see a healthy and steady growth in your SEO ROI. Those were my two cents my friend. Good Luck to you.

Best regards,

Devanur Rafi

Hi Andrew,

The best practice would be to come up with search engine/user friendly new URLs each corresponding to the month/year blog posts and then redirect them via 301. Essentially, you will be implementing a one-to-one or page-to-page 301. Redirecting a whole bunch of old pages to the homepage via 301 is not recommended especially if these old pages have some organic or referral traffic coming in. This would have a negative impact from your user experience perspective as well as from an SEO standpoint. A blanket 301 approach (bunch of pages being redirected to a single page, in this case the homepage) is the last resort that you should think about.

Please do not even think of letting those old pages giving up an HTTP head status code 404. This is a big NO, NO.

If I were you, I would have come up with new pages (URLs) for each of those old pages and redirected those via 301 to the corresponding new ones.

Best regards,

Devanur Rafi

Hi,

Having multiple pages with similar or identical content confuses the search engines and the outcomes in SERPs will be undesirable. Here is the deal: No two unique URLs should serve substantially similar or identical content. If it is the case, you should decide the URL that you would like to rank in the search engines, make others point to it via rel=canonical attribute. In general, the page that targets the most search keyword/phrase can be made the canonical URL or the preferred URL.

If I were you, I would have added unique content to the existing pages targeting the main keyword for the page.

For example, if the page talks about, 'used cars', this would become my target term for the page:

abc.com/cars/used-cars.html

I would also go ahead with a thorough keyword research & analysis to find which keywords/phrases are being searched more in your geo-location or target market, add corresponding pages with highly targeted content for each of these keywords/phrases (if not added already).

The key here is, content that is unique, up-to-date, highly relevant and useful to the visitors. Such content would bring in dramatic improvements to your overall SEO ROI and search engines like Google love such content and these pages will be awarded good positions in the SERPs going forward. As you know, high quality content is a natural link magnet.

Here is an action plan, if I were you:

1. Make these pages unique by adding unique content

2. Do a thorough keyword research & analysis to find new content opportunities in your niche

3. Add new pages with unique content based on the outcome of step 2.

4. Update the Sitemap.xml file and submit it to webmaster tools

5. Repeat steps from 2 to 4 once in 6 months based on the results or as and when required.

Those were my two cents my friend. Good Luck.

Best regards,

Devanur Rafi

Hi, with rel=canonical in place, there is no way that both the pages from the two sites appearing and ranking in the search results. Only the canonical or the preferred page will be indexed and can rank in Google.

You should be thinking along the lines to make the content on both the sites unique. Though these two sites operate and target the same niche, you can definitely make the content unique from each other.

Best regards,

Devanur Rafi

Hi,

While its not recommended to have duplicate content on your pages that is found else where, it is also not a good thing to de-index pages from Google. If I were you, I would have tried to beef-up these duplicate pages a little bit with unique content or at least rewritten the existing content so that it becomes unique.

Please go ahead and initiate the task of rewriting the product descriptions in phases starting with the ones that get the most traffic as per your web analytics data. Those were my two cents my friend.

Best regards,

Devanur Rafi

Except for very exceptional cases, Google considers and respects the rel=canonical implementation and its a strong hint for them.

Here you go for more:

http://www.mattcutts.com/blog/rel-canonical-html-head/

Best regards,

Devanur Rafi

Hi, the suggestion was not to get the quality articles written that take an hour to write each but I meant to change the products descriptions that were copied and pasted with little variation so that they don't look like a copy, paste job.

Now, coming to the de-indexing part, let us look at a scenario:

Suppose I built a website to promote Amazon products through Amazon associates program. I populated its pages using Amazon API through a plugin like WProbot or Protozon. In this case, the content will be purely scraped from Amazon and other places. After a while, I realize that my site has not been performing well in the search engines because of the scraped content but haven't seen any penalty levied or manual action taken. As of now, I have about 3000 pages in Google's index. Now I want to tackle the duplicate content issue. This is what I would do to be on a safer side from a possible penalty in future like Panda:

1. First, will make the top pages unique.

2. Add, noindex to the rest of the duplicate content pages.

3. Keep on making the pages unique in phases, removing the noindex tag to the ones that were updated with unique content.

4. Would repeat the above step till I fix all the duplicate content pages on the website.

It greatly depends on the level of content duplication and few other things so, we will be able to suggest better if we can have a look at the website in question. You can send a private message if you want any of us to have a look at it.

Hi William, thanks for sharing your experience here. My experience has been totally different from that of your's.

Here is what Dr.Pete has to say..

Taken from: http://moz.com/blog/rel-confused-answers-to-your-rel-canonical-questions

2) Can I Use Rel=Canonical Cross-domain?

Yes – in late 2009, Google announced support for cross-domain use of rel=canonical. This is typically for syndicated content, when you’re concerned about duplication and only want one version of the content to be eligible for ranking.

(3) Should I Use Rel=Canonical Cross-Domain?

That’s a tougher question. First off, Google may choose to ignore cross-domain use of rel=canonical if the pages seem too different or it appears manipulative. The ideal use of cross-domain rel=canonical would be a situation where multiple sites owned by the same entity share content, and that content is useful to the users of each individual site. In that case, you probably wouldn’t want to use 301-redirects (it could confuse users and harm the individual brands), but you may want to avoid duplicate content issues and control which property Google displays in search results. I would not typically use rel=canonical cross-domain just to consolidate PageRank.

Best regards,

Devanur Rafi