I've had to deal with a few clients that have been spammed by someone. In 3 out of 4 of these websites, it has had no effect on our rankings because we had a strong back link profile anyway. However, with the other, as it was a new website, these links ended up accounting for about 30% of the total links which ended up having an adverse effect on our rankings.
I'd disavow all the links and then try and get in touch with the webmasters of the website. I know you are meant to try and get them all taken down first but you should do both because it could take you months to get them taken down (if you can get a response at all). I take it you've not had a message in Webmaster Tools? If not then its an algorithm penalty rather than a manual penalty. I'd use as many tools as possible, Webmaster Tools, Open Site Explorer, Ahrefs etc to get a comprehensive list of links to your website and then start disavowing them....just be 100000% sure they are bad links!