When the big update came through a little over two years ago about page speed, I was working on a site with load time issues. I did see quite the drop in organic referrals from Google after that update was pushed, while organic referrals from Bing and Yahoo continued to increase. I was working in-house at the time, so I don't have a lot of other sites to judge at that time, but I did see distinct change in that one site.
I also agree with you that load time affects users and their browsing and conversion behavior. And even though "everybody has broadband" there are a lot of people on poor public (or conference) wifi connections or on their phones, and pagespeed is quite important to them.