A directory is just that. A directory; if someone has used your directory to find a site they are looking for then great! Google won't see it as a bounce, because they'll be coming to your directory and following one of your links, which isn't a bad thing.
If you wanted to make more of a resource for your users, you could provide rich descriptions on the sites in your directory (this may also help reduce spam submissions). You could even create your own rating system for the sites. This may create a better experience for your users as they would be able to search your directory and get some useful and insightful information about the sites they are visiting... This increases the amount of text and information on your site, meaning you have a better chance of ranking and may set your directory apart from others in your arena that just provide links to other sites.

