Welcome to the Q&A Forum

To answer your question directly: don't worry about it. I mean that both ways. Don't worry about the fact that you've not linked out to a lot of sites in the past. Google's not going to drop you down simply because you don't have a lot of outward-pointing links. Second, much like Chris said, don't make a point of avoiding external links.

Thanks for your replies, they were very helpful. After watching and reading I have decided that I need to implement rel="next" and rel="prev" in such a way that we handle multiple filters (facets) and sorting options so "to essentially pretend that only one version of the order/sort variable exists in every situation, and knock out the rest", that way Google will pickup rel=next sets for each facet on its own. The video made it clear that big view-all pages aren't great if there is a chance they will take time to load.