This is a good and common question. There are two reasons why you would want to keep the 301 redirect in place
- Usability: Users are still trying to reach this now defunct page
- Inbound Links: There are external links that are pointing to this page.
The best thing to do is first narrow down the pages for which you have no reason to keep 301s in place. Open up your analytics platform and look to see which of those 45K pages have received no traffic in the last 30 days. Then use a tool like Open Site Explorer or Google Search Console to see if any of those 45K pages have inbound links. Chances are, most of those pages neither receive traffic nor have inbound links. Go ahead and drop the 301 redirect for all of those.
Then, prioritize the remaining pages. The ones with the most links and most traffic should probably retain a 301 redirect for the long run. Those near the bottom you might choose to drop.
Finally, with that many 404s, make sure that your 404 page is useful. Create a custom 404 page that helps direct users to their correct location or provides an easy search mechanism to find it themselves.