Welcome to the Q&A Forum

lol well that would be a niche where you'll get some link troubles. But yes you're right it's very likely that some of those outgoing links are causing the issue. Any site that is spammy or in a spammy neighbourhood needs to be removed. Do you really need to keep any of them? Reciprocal links really don't do a lot for you anyway.