I would look at them for the entire backlink profile, but pay more attention to the followed links when it comes to something like anchor text distribution. Here's why:
If I were Google and I were looking at a car insurance website, and I saw that it had 200 followed links with brand mentions, URLs as anchor text, "click here", etc., then that looks fairly good. However, if the site also has 400 links with "cheap car insurance" as anchor text, does that look somewhat like the site has been doing something like comment spamming? Yup, it does. They make up a telling part of the profile and Google would be silly not to take them into consideration, but they should not directly affect site authority in the way followed links do.
Keep in mind that DA scores are a Moz metric and whilst Moz attempts to replicate how Google operates to the best of its ability, it isn't a Google figure so can't be relied upon to show how Google views a site's authority.
If you've been penalised, I'd most certainly start with removing followed links that are poor in quality. I have not heard of a situation where someone has recovered from a penalty after removing nofollowed links (this doesn't mean it hasn't happened). It's the followed links that Google cracks down on if they dislike them, and the followed links that make a large difference to your rankings.
