I understand your fears completely because WP + themes + plugins can be HUGE mess.
Why? Because some of devs don't know technical and on-page SEO. I have seen themes where they put text "Comments" within H1 tag. At same time post title was encapsulated with H3. Things as hidden text, messy HTML codes, bloatware HTML are countless. You also can get server overloading, slow SQL queries and some issues. Can be performance issues, PHP issues, hosting issues. Running WP mean that you will get 10 CSS files and 10 JS libraries. And this is "best case scenario". Just imagine what is worst. Now add "upgrading" procedure where everything can be broken (or changed!) with just one click. Ah, and mine favorite - security exploits and hacking. Sound like "perfect storm". Isn't?
Now i know that this sound scary. That's why you should see and review HTML code of WP before migration. Probably you will see potential for improvements. And this changes need to be patched over original files (i'm talking about theme or plugins). For theme is OK - you can make child theme based on original. But for plugins - you need to "fork" original plugin and make your own custom version. Then on each update you should "diff" your version and original to keep your patches and new code. This mean very strong backup solution plus local dev environment and extra work on each update.
Also - it's year of 2016. Why you don't around for alternatives? I can give you suggestion - static site generators:
https://www.smashingmagazine.com/2015/11/modern-static-website-generators-next-big-thing/
https://www.smashingmagazine.com/2015/11/static-website-generators-jekyll-middleman-roots-hugo-review/
As you can see i'm not giving answer Yes or No. I'm just giving you few extra points to think about. Just put cards on the table.
PS: May sound negative little bit because i have some theme and plugins in past. One wrong choice and all SEO efforts can be ruined. Such is life...
Glossary:
"fork" - process of creating different version of something existing with some changes that doesn't exist in original. You can hear that devs are forking projects too. In your case - since you can apply your patches but on next update everything will be gone back to original. That's why you need to fork them.
"diff" - process for checking difference between files/project and extracting/showing only difference between them.