Welcome to the Q&A Forum

There is data out there that suggests Google updates rankings for less competitive keywords less often. So even though your backlinks may have disappeared a long time ago, its possible your position in the SERPs is only now being adjusted. Its also possible that you or sites linking to you were hit by the panda update. In terms of what to do about it, I wouldn't advise going "wild" with backlinks, like you say above. Use some of the strategies you can find here on SEOmoz to develop quality backlinks and a good user experience.