To noindex or not to noindex

[GreatFire.org]

Our website lets users test whether any given URL or keyword is censored in China. For each URL and keyword that a user looks up, a page is created, such as https://en.greatfire.org/facebook.com and https://zh.greatfire.org/keyword/freenet. From a search engines perspective, all these pages look very similar. For this reason we have implemented a noindex function based on certain rules. Basically, only highly ranked websites are allowed to be indexed - all other URLs are tagged as noindex (for example https://en.greatfire.org/www.imdb.com). However, we are not sure that this is a good strategy and so are asking - what should a website with a lot of similar content do?

1. Don't noindex anything - let Google decide what's worth indexing and not.
2. Noindex most content, but allow some popular pages to be indexed. This is our current approach. If you recommend this one, we would like to know what we can do to improve it.
3. Noindex all the similar content. In our case, only let overview pages, blog posts etc with unique content to be indexed.

Another factor in our case is that our website is multilingual. All pages are available (and equally indexed) in Chinese and English. Should that affect our strategy?References:https://zh.greatfire.orghttps://en.greatfire.orghttps://www.google.com/search?q=site%3Agreatfire.org

[AgentsofValue]

Creating a page every time someone performs a search could probably spiral out of control pretty quickly.  If you have a certain amount of 'page rank', based on all of the back links you have, that page rank will get shared to all of the pages that you have across your site.

One way you could more naturally control what gets indexed, is by what you link to from your home page.  For instance, if you track the most blocked big sites, as well as the most blocked keywords, and have those pages 1 link from your homepage, you could expect those to get indexed naturally when your site is spidered.

As you get more links from other sites, and your trust from the search engines and page rank grows, you should be able to support more pages getting indexed across your site.

There is the issue of your site contents potentially being regarded as 'thin content', since many of the pages appear to be the same from page to page.

One question I had - I saw your site hosts both Chinese language words and English language words, and checks whether those words are being filtered.  Perhaps it would make more sense to only show the words in Chinese characters on the zh. subdomain, and the English words on the en. subdomain?  Just a thought.  Is there any difference between the zh and en subdomains, aside from the language of the template?

Really interesting website.

[MoosaHemani]

Ok I might sound completely stupid here as I never come across this case before but here is my hypothesis….

While searching for a keyword or URL you another field (may be a checkbox) that represents the category of search.

So, ones the new URL will generate it will come under the specific category automatically.

Customize the category pages so that they look different from each other.

Index the category pages and add canonical tag on any new generated URL of the category page. For example if the new page generates like www.yourwebsite.com/movies/ice-age -3/ this page should have the canonical tag to http://www.yourwebsite.com/movies/

Why?

Creating category pages will allow you more unique pages to get indexed in SERPs without the duplicate content issue. Adding canonical tag on all other URLs will tell category pages are the real pages that Google should consider.

This might help you cater more chances to earn more search traffic from Google.

**This is my assumption what I think should work!

[GreatFire.org]

Thanks for suggestions. I have some follow-up questions. Would really like to know what you think about the following:

1. The "page rank will get shared to all of the pages that you have across your site". In general, does this mean that if I add noindex tags to all but a few pages, they will be ranked much higher? Currently thousands of pages are indexed. Is it correct to say that if only say 20 pages were indexed that would greatly improve their ranking?
2. The zh and en versions of the website have different templates and most of the text content is also translated (with the main exception of old blog posts). We could add noindex on all of the zh website or all except the main pages. Would you recommend that?

[GreatFire.org]

Hi Moosa. Thanks very much for your reply and great suggestions. If I add canonical tags on each URL page referencing the category page where it belongs, should I also add noindex tags on it? Should then actually all URL pages have noindex tags and only allow category pages to be indexed?

[MoosaHemani]

Nops, no need to include the no index tag as adding canonical is an indication to Google that what are the original pages that search engine need to index and crawl so al other pages then category pages will be crawled automatically.

[AgentsofValue]

1.  yes - if you no index all but 20 pages, those 20 pages would get a boost in rankings.  You would end up losing the long tail searches from those other thousands of page - so you'll need to do some cost / benefit analysis on that.

2.  you'll need to do a cost / benefit analysis on this one.  Are most of the visitors to your site searching in Chinese or English?  Are your search terms mainly in Chinese or mainly in English?  Are your Chinese speaking visitors more likely to want to visit the .zh subdomain?

You could publish 20 to 50 pages on each subdomain, and then focus on doing some link building.  If you have strong rankings across those 40 to 100 pages, then you could start adding more pages slowly over time.