It depends if yours is a high authority site or not. If its a relatively new site or with low-avg authority, it could very well have fallen under the 'fishy' side of the radar.
Its unlikely that nofollow links would trigger a penalty but the # of links you mention here is staggering so I wouldn't count that out.
Also, Google made 40 changes 5-6 days ago, which is when your saying your site traffic fell so it could be that as well: http://insidesearch.blogspot.com/2012/02/search-quality-highlights-40-changes.html