Abnormal crawl issues appearing in my Moz results

[Mutatio_Digital]

I have been asked to look at a site for a friend and was more than surprised to see 16,9k crawl issues appear in the dashboard... of this 6,238 are duplicate page content and 5878 are duplicated page titles.

What on earth is going on? I have spoken to the web developer as it appears there is a dev site somewhere and this is his response

[Can I stress that Google determines which site was in the index first and then removes other sites it sees as having duplicate content.  Our dev sites appearing in the search index would not affect your ranking due to duplicate content as Google would see your site as the first site with the content]

As I cannot make contact with him, I am scratching my head, surely a dev site should be no-indexed, it sounds as though he is saying that its ok because Google will take the main site as the first site with the content...

Very confused! Help need MOZ community.

Manythanks,

Sarah

[DirkC]

Moz is only indexing pages it's crawler is able to find. This implies that on your production site you have links to your development site.

Don't really agree with what your dev is saying - he should correct these links first; put a noindex on these pages. Alternative - put a password on the dev site so it's only accessible with a password. If a lot of users are putting links to your dev site it could become more important than your main site. Google will try to choose the most appropriate site - but you have no guarantee that it will choose the right version. In any case - that's not the type of risk you should be willing to take.

Once this is done - you can request a removal of these pages via the search console.

If all pages are removed from the index you can adapt the robots.txt to prevent access to the Google & other bots. Do this only after all pages are removed - if not Google will never find the noindex directive.

Dirk

[Mutatio_Digital]

hello Dirk, thank you for that great answer, we have since been doing a bit more digging of our own and before we go back to the web developer we want to check what should be happening with the links the we are finding duplicated as we are seeing that the issues relating to Duplicate Pages are coming from links from the login page which shows information about where the user was redirected from.

For example, if the visitor is not logged on and wishes to wish-list an item, they will be redirected to the login page, with the item code and intended action in the url; which can then continue on to the desired page once logged on.

The MOZ crawler is seeing these pages as having Duplicated Content whilst they are all the same apart from a piece of information in the URL. Should we be blocking these duplications? Are they a risk to us? What should we be doing?

I have also added this as a new question - I am quite new to this community thing so wasn't sure which was the best way to ask the question.

Many thanks again,

Sarah

[DirkC]

Hi Sarah,

Googlebot will follow these links as well and discover these "useless" pages (the are off course not useless from human perspective but they don't add value for bots - and they will be considered as duplicates). Duplicates are no reason for "punishment" - so you could just let them be. Personally I would put a nofollow on these links or add a "noindex" tag to the login page. Normally you shouldn't use nofollow on internal links - but login pages are an exemption on this (check also https://searchenginewatch.com/sew/news/2298312/matt-cutts-you-dont-have-to-nofollow-internal-links : "Of course, there are always exceptions to the rule, and things like login pages can be the exception. He said it doesn’t hurt to put the nofollow link for a link pointing to a login page, or things like terms and conditions or other “useless” pages. However, it doesn’t hurt at all for those pages to be crawled by Google."

For the practical part - if you add an additional question to a question which has been marked as answered - only the ones who have already answered will see the additional question. To be on the safe side  - it's better open a new question if you want other people to have a look at it.

Hope this helps,

Dirk

[Mutatio_Digital]

Thanks again Dirk. I like your direct and knowledgeable responses. I have sent a Linkedin connection!!

Many thanks,

Sarah