Welcome to the Q&A Forum

Hi,

You can look at your Google Webmaster Tools account to know what are the search queries that your visitors used to land on your website from Google. You can also look at your web analytics data (like Google Analytics, SiteCatalyst etc) to know this data but as Google started encrypting the search, you won't be able to see the complete picture of your keyword data that is drving traffic from Google. If you don't have an access to webmaster tools account or web analytics, you can try using tools like SEMrush to have a fair idea about the keywords that are driving organic traffic to a website.

Coming to the strategies, useful, relevant, updated content stands at the top followed by backlinks coming from other topical niche sites, good blogs and forums, authority websites like .gov, .edu, .org so on and so forth. If you can come up with top notch content, half the battle is won as good content is a natural link magnet. Those were my two cents my friend. Good Luck and please post back in case you have any other queries in this regard.

Best regards,

Devanur Rafi

Hi,

Pages blocked by robots.txt file will not be considered for ranking. You still see those pages in Google's index is because of their mentions on other websites. Google has been doing link discovery from websites other than these Web2.0 sites of yours. So to stop Google completely from indexing these pages, you should go in for page-levelrobots meta tag. Here you go for more: https://support.google.com/webmasters/answer/93710?hl=en

Once you implement the page-level robots meta tag to block those pages, Google will remove them from the index after the next crawl. You can expedite this process by using the link removal procedure in webmaster tools account.

Here you go for more: https://support.google.com/websearch/troubleshooter/3111061?rd=1

Good Luck to you my friend.

Best regards,

Devanur Rafi

Hi,

The redirection implemented here is JS based with a delay of 10 seconds, the HTTP header status throws a 404 status code, so the status will still be 'Not Found'. Moreover, for the crawler, the page does not get redirected as it cannot process JS so, for it, the status will be, 'Not Found'. You can check this with Screamingfrog SEO Spider. You can even try to disable JavaScript in your browser and you will find that the page does stays as is and will not redirect to the homepage.

Apart from this, your non-www version of the URL (http://wearyouwant.com/en/brand-bisous-bisous.html) is being redirected to the homepage via 301. This should not be the case. The non-www version should have been ideally redirected to www version via 301.

The ideal way should be, to handle the 404 error using a custom error document and then give an option to the visitor to decide how to go about your website. You should not automatically redirect the page.

Good Luck my friend.

Best regards,

Devanur Rafi

Hi,

While no single backlink checker on this planet would show you same amount of backlinks that a website has, the only better way is to consolidate data from different tools like, ahrefs, OSE, Google webmaster tools account etc. No single tool can give you comprehensive backlink data.

Hope it helps my friend. Good Luck to you.

Best regards,

Devanur Rafi

Hi,

I agree with Dennis here.

One question that you can ask yourself:

"Why should Google have two websites with substantially similar content in their index?"

So, the answer would be to come up with substantially unique content on both the websites so that both of them can pass Google's radar safely  and justify their presence in their index.

In the past, I worked with a client who had multiple websites operating in the same niche but catering to different states in the US. With almost identical content on many of them, they were not at all ranking good in the search engines overall. We had the content rewritten in such a way that all of them had unique content (please note that all these domains still operated in the same niche but with slightly different keywords/phrases based on the differences in naming conventions in different states). This worked wonders. The results exceeded all the expectations of the client. There was a dramatic increase in the total number of pages from these domains in Google's index along with a whopping increase in overall ranking for many of the competitive terms in the search engines.

With substantially unique, updated, highly relevant and useful content, you will see a healthy and steady growth in your SEO ROI. Those were my two cents my friend. Good Luck to you.

Best regards,

Devanur Rafi

Hi Surabhi,

Its because of the content (the keywords and phrases) on your homepage that your theme is not being recognized as, 'educational'.

Here is something that you can do:

1. Install SEOQuake plugin for firefox.

2. Open your homepage and your competitors' homepages in different tabs.

3. Right click on your homepage > SEOquake > Keywords density  and look at the keywords on your homepage and do the same for your competitors' homepages. Compare them with that of yours. You will have your answer.

Almost all your keywords/phrases mention about career and not about education or different courses for study.

Hope you get what I wanted to convey. Here is what you can do. Try to do a small competitor analysis and study their content. You would need to incorporate terms that are tightly related to the seed term, 'education'. You should use some LSI (Letant Semantic Indexing) keywords in the content.

Good Luck to you.

Best regards,

Devanur Rafi

James got it spot on. Google automatically displays site links.

Here is what Google has to say in this regard:

"We only show sitelinks for results when we think they'll be useful to the user. If the structure of your site doesn't allow our algorithms to find good sitelinks, or we don't think that the sitelinks for your site are relevant for the user's query, we won't show them.

At the moment, sitelinks are automated. We're always working to improve our sitelinks algorithms, and we may incorporate webmaster input in the future."

Here you go for more:

https://support.google.com/webmasters/answer/47334?hl=en

Best regards,

Devanur Rafi

Pleasure is all mine my friend. You are most welcome. Moz SEO community is an indispensable asset and weapon in any SEO's inventory in my opinion. We learn a great deal here while helping others. I am really thankful to each and everyone here on Moz community. Long live Moz and Mozzers. YOU ROCK!!

Yes, of course my friend, Google has to crawl the page to see the page-level meta robots tag but till date I have not seen any page in Google's index that has been blocked using the robots.txt file and page-level meta robots tag. Password protecting your .htaccess file would be an overkill if you just want Google not to index a page. If you want Google to remove any particular page from its index, you can get it done from webmaster tools account. Here you go for more: https://support.google.com/webmasters/answer/1663419?hl=en

Good Luck to you my friend.

Best regards,

Devanur Rafi

Hi friend,

If a page has been blocked using Robots.txt file, then Google will not crawl and index the page from within the website but what if a reference of that page is found on a third-party website? In cases like this, link discovery will happen and the page will be indexed without a Description snippet and such pages will have the following text in the place of a description in the search results pages:

"A description for this result is not available because of this site's robots.txt – learn more"

So inorder to completely stop Google from crawling and indexing a page, you should should block the page by implementing, page-level meta robots tag.

Here you go for more: https://support.google.com/webmasters/answer/156449?hl=en

Please feel free to post back if you have any other queries in this regards.

Best regards,

Devanur Rafi

Hi, while it is definitely possible for Google to penalize a page or a folder for that matter any section of a website, unless we take a look at the page you are talking about, all that we write will be just shooting in the dark.

Best regards,

Devanur Rafi.

Hi, you can try using Gsitecrawler.

Download it from here : http://gsitecrawler.com/en/download/

Best regards,

Devanur Rafi

Hi William, thanks for sharing your experience here. My experience has been totally different from that of your's.

Here is what Dr.Pete has to say..

Taken from: http://moz.com/blog/rel-confused-answers-to-your-rel-canonical-questions

2) Can I Use Rel=Canonical Cross-domain?

Yes – in late 2009, Google announced support for cross-domain use of rel=canonical. This is typically for syndicated content, when you’re concerned about duplication and only want one version of the content to be eligible for ranking.

(3) Should I Use Rel=Canonical Cross-Domain?

That’s a tougher question. First off, Google may choose to ignore cross-domain use of rel=canonical if the pages seem too different or it appears manipulative. The ideal use of cross-domain rel=canonical would be a situation where multiple sites owned by the same entity share content, and that content is useful to the users of each individual site. In that case, you probably wouldn’t want to use 301-redirects (it could confuse users and harm the individual brands), but you may want to avoid duplicate content issues and control which property Google displays in search results. I would not typically use rel=canonical cross-domain just to consolidate PageRank.

Best regards,

Devanur Rafi

Except for very exceptional cases, Google considers and respects the rel=canonical implementation and its a strong hint for them.

Here you go for more:

http://www.mattcutts.com/blog/rel-canonical-html-head/

Best regards,

Devanur Rafi

Hi, with rel=canonical in place, there is no way that both the pages from the two sites appearing and ranking in the search results. Only the canonical or the preferred page will be indexed and can rank in Google.

You should be thinking along the lines to make the content on both the sites unique. Though these two sites operate and target the same niche, you can definitely make the content unique from each other.

Best regards,

Devanur Rafi

Hi,

Its not new that Google changes the page titles and descriptions.

Here you go for more:

http://searchenginewatch.com/article/2342232/Why-Google-Changes-Your-Titles-in-Search-Results

and here..

http://searchengineland.com/google-title-wrong-157819

Best regards,

Devanur Rafi.

Hi Mike, this is awesome but strangely, the only page on Wikipedia that gets returned for the following query,

"inurl:digitalization site:wikipedia.org" in Google is <cite class="_Zd">ar.wikipedia.org/wiki/Digitalization</cite> that too in Arabic language. So they do not have the exact match definition in English at this point in time as I write this. You can confirm this with the following query:

"inurl:digitalization site:en.wikipedia.org"

Out of the first 10 results in Google for the query, "inurl:digitalization", I tried to run the following query for each of these top 10 websites:

"inurl:digitalization site:<each 10="" of="" the="" top="" domain="" names="">",</each>

Only Gartner.com has 26 URLs (with "digitalization" in the URL) with only one exception, digitalization.nl which can be ruled out as the domain name itself contains, "digitalization". This case can be dug much,much deeper but let me stop here.

Without having dug deeper in to this case, at this point, the only thing I believe would replace Gartner's definition with the definition from Wikipedia is, if they come up with a page that contains the term, "digitalization" in the URL and in the title.

Best regards,

Devanur Rafi

Hi, its time to check your titles and descriptions. With all respect for Yoast plugin, I would still recommend to write the page titles and descriptions manually at least for the most important pages on your website. Google changing the titles and descriptions is not new and all that we can do is try and test different titles and descriptions.

Here you go for more on this..

1. https://yoast.com/google-page-title/

2. http://searchenginewatch.com/article/2342232/Why-Google-Changes-Your-Titles-in-Search-Results

3. http://wordpress.org/support/topic/plugin-wordpress-seo-by-yoast-title-changes-to-individual-pages-categories-tags-posts-is-not-working

and here..

4. https://productforums.google.com/forum/#!topic/webmasters/K6D14caef7U

Hope they help.

Best regards,

Devanur Rafi

Hi friend,

You can use SEMRush for the purpose. You can have the data that you were looking for under,

Organic Search > Positions

There you can see all the popular URLs (pages) from a domain sorted as per the total organic traffic that they each receive. You can sort using many parameters there.

Hope it helps. Good Luck to you.

Best regards,

Devanur Rafi.

Lesley is absolutely correct. Writing a unique copy or even thousands of unique variations of a single article is very much possible. The best examples are article directories that contain thousands if not lakhs of unique articles on a same topic. Writing unique copy does not always mean to re-invent the wheel but can also be rewriting it while retaining the meaning or essence of the article, including some phrases or sentences unique to the author's writing style or something of that sort.

The key here is to make your copy different from millions of others that are already written and in the search engines' indices. It is also highly important to come up with such a variation that really adds value to your website and the Internet on the whole. Your web copy should justify its presence in the search engines by providing real value to the visitors. If you can do this, chances of your copy out-ranking the competition become high (of course can depend on ton of other ranking factors) as such content will be a natural link magnet and within no time, it would gain lot of SEO goodies.

Being a very strong advocate of unique and quality content my self, I give clear instructions to my content team while pointing them to few sample articles that are already there on the Internet.

Best regards,

Devanur Rafi.