The first question you should be asking yourself is "why did our competitor lose his rankings?"
The answer would be something like:
- they acquired too many links too quickly
- anchor text profil isn't natural
If one of the reason was that they got links for site A, B and C that you too have links from, then you would probably have dropped too!
My point is, it is not about the websites you are having links from, it is all about the global link profile and the time taken to acquire those links.
Now, you are also wondering if you should preventively remove those links, my answer would clearly be NO, but it you ever decide to do so, please do it slowly so you don't hurt your rankings too much and try to compensate with other natural links progressively.
And last but not least, we always tell our cutomers to concentrate on getting new links that correct their link profile rather than waste time and effort trying to change things they already have.
Hope that helps!