Hi
It definitely makes sense to clean up your domain list, an extensive domain list is often time consuming to maintain and frequently the majority of the domains deliver little or no value.
I would personally take the following approach:
1. Is the domain worth having and does it represent any value to the business?
A: yes - keep it
B: no - go to question 2
2. Does the domain offer any benefit to the site in terms of link juice/domain auth/page auth etc (I would check this thru Open Site Explorer, Majestic and Ahrefs if you have access to them)
Yes - Keep it
No - got to question 3
3. Is the domain potentially harmful to the site?
.xxx domains are indexed by Google, enter the following into the Google search bar "site:*.xxx" and you will see plenty of sites. My concern would be .xxx site are targeted to porn industry, if your website has no relation to this industry Google could decide to penalise you for this in the future, so i would be very tempted to ditch/park all the .xxx domains unless there is a good reason for keeping them.
Ultimately any decision to ditch domains should be taken with caution, as once you loose a domain, you can't get it back cheaply or easily.
I hope that helps
Justin