@JustDucky We recently migrated to a data center and the average loading time dropped from ~4 seconds to ~0.9. I to noticed only 1-2% drop in bounce rate. It seems only that many people were turned off by the loading times. Then again 1-2% can be anything.
@John O'Haver I would invest the time simply because ~3.4 is the average value. This means that sometimes it goes up to 10 or even more. Take a look at your analytics account and see the performance per country. Also, I've been benchmarking analytics with remote monitoring solutions and I find a discrepancy of about 30% (probably due to limited sample date from analytics). I don't want to advertise any available solutions, but trying one won't hurt. You may find your times to be better (I hope).