Hello,
Yes it does hurt your SEO in terms of creating an authoritive web site. Wordpress is an over-optimisation nightmare if you leave the default setting on. It is a small issue you can solve by changing some of the setting in Yoast SEO plug in. have you installed this plug in? If not, do it and once you have done this do the following:
On the general setting: Tick the box "Noindex" on the sitewide meta setting
On the post type setting: tick noindex/follow on the Meta Robots under the MEDIA section
On the taxonomy setting: tick noindex/follow on the meta robots under the TAGS section
On the author archive: tick noindex/follow and Disable author archives under the author archive section. Tick ONLY the noindex/follow under the data archive section
On the XML SITEMAP setting:
tick media on "Exclude post types" - tick tags and format under the Exclude Taxonomies section.
You should be ok after that!
ps. by tick I mean select 
Hope this helps!