Hi Jonathon,
I would agree with James (oh and Ryan too!) that the sudden addition of that number of links may alter your link profile in such a way as to signal to search engines that there may be some kind of manipulative link building practice going on.
Something that is likely to make this appear even more like a manipulative practice would be if the Lion's share of the anchor text used to create the links to your site is identical, for an exact match keyword term etc.
One thing we should clarify is that while this may be having a negative effect upon your rankings, it is not really a "penalty" as such. Basically, the algorithm is seeing a sudden and unnatural looking change in your link profile and adjusting your ranking as a result. If Google has manually applied a penalty for any reason, they are now informing webmasters through notifications in Google Webmaster Tools. Manual penalties usually only happen if you are doing something really overt like buying or selling links. As Ryan mentioned, if the directory is part of a paid link system, it would be an issue. In that case I would expect to find a message in Webmaster Tools.
If it were me I would leave it be for a while and see what happens.
First, provided the directory is not a problem, with 20,000 targeted referral links out there, your potential for increased conversion ready traffic is greatly enhanced anyway.
Second, I would expect that with a little time and maybe some more gradual and diverse links added to your link profile (and no more major spikes), your rankings will tend to come back anyway. If they don't begin to improve (or continue to worsen) within a month, I would bring it to Google's attention by filing a Reconsideration Request. This link is the latest from the Webspam Team on the current process.
Hope that helps,
Sha