You are 100% right. That's it. The concern is having 100's of thousands of links that would be very very low quality links. The affiliate link pattern will make it "sort of okay" if you decide to keep them follow. Those links are there for the affiliate tracking, but typically the majority of the web uses some sort of an affiliate tracking site like GAN (Which is closing down), CJ, Linkshare etc. Considering that, is it really worth the risk of having that many number of links ? Maybe, maybe not. And that's the business decision you need to make.
The affiliate traffic and sales is important (which justifies the affiliate program). Your natural SEO rankings (current and future rankability) helps you justify the importance of these kinds of decisions and which is why you are asking this question.
We just need to find the right balance without tripping too much on either side.
