Sure, there are two areas to address on your server config. One is the cache-control which is returned in the header. I set a longer period of cache for all images and scripts to save users downloading new copies time and again to their cache. ie: max-age=3600, must-revalidate - see more here: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
Then I would also set some rules around using 304 status on page furniture and other assets which do not change frequently.
Aside from this, as far as I am aware, you should ensure your stack is optimised. The recent Search Insight session from Google was interesting as in that presentation they talked a lot about the average load and latency times they see, useful to use as a benchmark in tuning your own speed.
Cheers,
Damien