If they are causing you to have duplicate content and there isn't much search traffic going to them, there should be no penalty for removing them. 404s don't ever cause penalties. You can tell Google to stop crawling them though, but it is a time intensive process that is probably not worth your time. https://support.google.com/webmasters/answer/9689846?hl=en
If you are deleting and then remaking the tag pages in a different URL structure, I would note which ones are getting organic search traffic and are most used by site visitors. Then when you remake them, put in a 301 redirect from the old URL to the new one.
