Welcome to the Q&A Forum

You don't seem to have much options if canonicals are not possible.

To be honest - in cases like this I wouldn't use canonicals but always go for 301 redirects. They are pretty easy to implement (2 lines of code in htacess) and they are directives Google has to follow (and will pass pagerank). Canonicals are mere requests to Google that it can but is not obliged to respect.

Would also indicate in the Search Console of the duplicates that you're moving them to the main domain (https://support.google.com/webmasters/answer/83106?hl=en) - you need to validate the duplicate domains before implementing the redirects.

Dirk

Try contact Moz support directly via the form https://moz.com/help/contact/local - they do monitor the Q&A but it's probably faster trying to contact them.

Dirk

There is long article on the dev blog how they determine whether pages are duplicates - check https://moz.com/devblog/near-duplicate-detection/ - it's quite technical stuff - but this is the part which might interest you:

"This leads to one of the questions we get asked a lot: Why do I see duplicate content warnings in the context of Custom Crawl for pages that I see as different. Ultimately, it’s always because of the same reason: because no dechroming is done, there is a small amount of unique content relative to the total content. One of the places where this crops up a lot is web stores, where there’s a large amount of chrome layout, but only a short product description associated with it."

Dechroming : removing things like navigation, footer, ..etc from the page (exact def. to be found in the article)

If you compare both pages - apart from the image & product title there isn't too much difference between them so the crawler sees only a very small % of content which is different and marks them as duplicates.

Dirk

Well it's quite difficult to judge the quality of the content as I don't understand what is written on the page. What I did notice is that each time I clicked on a link a new tab which advertising (https://trafficnado.com...) was openend (without me requesting it) or that a pop-up was blocked.

Google does consider this to be quite spammy techniques, so could be the cause why this page has suddenly disappeared from the results.

Not sure about the content - seems you are streaming series. Are you the copyright holder for these ? If not, this could also be the reason why your site did disappear from the results (DMCA complaint)

Dirk

Great it worked. Just a small note - if Screaming Frog is getting confused by all these parameters, it could well be that Googlebot (while more sophisticated) is also having these issues. You could consider to exclude the formkey parameter in the Search Console (Crawl > URL parameters)

DIrk

In the menu 'url rewriting' you can simply put the parameters the site should ignore (like date, formkey,..). I removed the formkey parameter and I checked the pages of the archive in Screaming Frog.

It is clearly able to detect all the internal links on the page - so will crawl them.

How are you certain that the pages below are not crawled - could you give a specific example of page that should be crawled but isn't?

Dirk

Bob,

It depends on the category & type of product. I remember a Dutch site selling shutters who just put the NL content on a BE domain - problem was that in Belgium we don't use this word when looking for this type of product and hence Google wasn't showing the site (they did rank pos. 1 for shutters in Belgium but probably with 0 traffic)

You don't have to rewrite the content for Google - but it would probably be a good idea to let a Flemish person check the content. If it's just a small word here and there it's no problem - if it's about your main keywords then it's an issue

To reply to your other question - when searching in BE I quite often get NL results if Google doesn't find a good BE result or the NL site is just better. You could just put the content on the be domain - and see if it brings results (even without doing the cross-linking - although I think that would be a useful feature). Belgian backlinks will always help - but it will take time & effort. Take a trial & error approach - there is no risk - if it doesn't work you can always improve later on.

Dirk

It means that his site is an exact duplicate of your site and could potentially outrank your site in the search result (unless there is a canonical on which is pointing to the correct url)

He just pointed his domain to your server - instead he should redirect to your site.

Dirk

It think Screaming Frog is going nuts on the formkey value in the url which is constantly changing when changing pages.

Could you modify the settings of the spider to respect noindex & respect canonical - looks like this is solving the issue.

Alternatively you could rewrite the url to ignore the formkey (remove parameter)

Dirk

Did you put in some special filters - just tried to crawl the site & it seems to work just fine?

I agree with Jordan on this - shouldn't cause troubles.

Just make sure that you at least adapt the wording on the site  - we might both speak dutch but not all the words have the same meaning & we don't use the same words to describe the same things. As an example - in Belgium we like "konfituur" - you prefer "jam" - pretty useless to try put a page optimised for "jam" in Belgium as nobody will look for it.

Dirk

Great video Matt! Should be 'obligatory' to watch this video before posting the first question.

Just to be sure - you want to present something like this:

IP Address = New York
Visitor is seeing a page

domain.com/results_for_new_york

with canonical domain.com/results_generic

This might work - but it's not really a correct use of a canonical url (which is intended for duplicate content which is not really the case here) - so not sure if Google is going to respect the canonical in this case (canonical = hint - not a directive)

Personally wouldn't do it this way.

Dirk

I would give the choice to the user rather than forcing him to a certain page. You could use the IP detection to trigger a message like "We noticed that you are from New York. Would you like to visit the <new york="" page="">or rather view the <generic page="">?". You could consider to store the choice in a cookie, so that on subsequent visits the visitor always goes to the New York pages rather than the generic ones.</generic></new>

This solution is both user friendly and doesn't have impact on SEO. If you use IP detection to force users to a certain page, there is always a risk (depending on the implementation) that Google is only indexing the Californian pages (as main Googlebot IP is Californian)

Dirk

It's not a big issue. Just mark as fixed - if they 404 & no links exist them they will disappear from the reporting (every site has 404 errors). Even if you don't do anything it's not a problem;

Dirk

You might want to check this thread https://moz.com/community/q/804-https-ssl-error - there are possible solutions mentioned in the answers if you scroll towards the bottom of the page

Basically Moz has issues crawling https sites using SNI - which is often use by Cloud Flare (and some other hosts)

"Our current crawl libraries do not support SNI which is typically what is enabled by default by Cloud Flare. Our engineering team is still investigating expanding our libraries but an ETA is not available quite yet."

As far as I know this issues hasn't been solved yet. If the SNI is not the issue - you might want to contact Moz directly (help@moz.com) to take a look at your specific issue.

Hope this helps,

Dirk

Normally in cases like your site Google is showing a message that the remaining results are similar to the ones.

Some reasons why Google is not displaying all results:

• You have a lot of semi-duplicate pages like http://www.blackbowchauffeur.com.au/theme-park/aussie-world-to-cedar-creek-from-cedar-creek-to-aussie-world/947583845769996550
• Some of your url's have a backslash / others not - in your internal linking you use both. The backslash version have non-back slash as canonical (not sure if this is always the case)
• You redirect url's with capitals to the non-capital version - but you use the version with capitals in your navigation
• pages like http://www.blackbowchauffeur.com.au/airport-transfer-connection-to-and-from-suburb-starting-with-h/ don't really add value for the index either
• you might also want to check your 404 pages - they look like a 404 - but non existing pages are redirected with 302 to 404 page with http status 200 - example http://www.blackbowchauffeur.com.au/price/lismore-airport-to-highfields-from-highfields-to-lismore-airport/faq -> 302 to http://www.blackbowchauffeur.com.au/404-page-not-found which unlike the page is stating has a http 200 status

Summary - after crawling 1892 url's of your site - my crawler found 1215 redirected url's (quite possible that these count as url in the results when you do site:mysite.com) as you continue to use them in your site's navigation. From the url's that are actually working a lot of them are near duplicates.

Use a tool like screaming frog to clean your internal links - point internal links directly to the final destination. Check if you really need all these detailed pages - they are just adding weight to your site and I am not sure if Google will index them properly + show them in the search results. Also correct the 4xx page - it should show 404 as http status.

Apart from all of the above - the site:mysite.com is not the most reliable way to check which pages are in the index. Check your search console (and the landing page report in Analytics) & combine this info with a log analysis to see which pages Googlebot is actually visiting. You will learn much more about issues with your site than using the site: command.

Dirk

Hi,

According to Google the value of the links is transferred to the https domain if your http is properly redirected to https.

Check this post on Google+ from John Mu: https://plus.google.com/+JohnMueller/posts/PY1xCWbeDVC

Quote: "Do I lose "link juice" from the redirects? No, for 301 or 302 redirects from HTTP to HTTPS no PageRank is lost. "

Hope this helps,

Dirk

To be honest - I would change web designer. Making your site responsive is all about reformatting your existing content to make sure the layout adapts to the used device. The main work is done in the css - not in the HTML.As far as I understand your question your webdesigner is duplicating the same content in the HTML, which normally shouldn't be done in order to make your site responsive.

To answer your question - the content will be appearing twice - once visible - once hidden depending on the device /type of Google bot. Google could consider this as spamming, or just ignore the part which is hidden, difficult to tell. You will in any case have a bloated HTML code which could have an impact on the load times of the page.

Dirk