Hmm, I am a big advocate of quality content and a good user experience with decent on-page SEO having a positive impact on rankings. However the majority of high ranking sites all have links - after all, they are a vote of confidence in the eyes of Google (obviously a very broad statement and not all links are equal or help). If your site is that good and lots of people visit it you would expect to earn links whether you actively go after them or not. In order to help yourself why would you not try to earn links, I would say, even if a low competition term.
In relation to your original post will it take longer to rank? There are lots of factors that can impact this but I would say it will definitely take you longer to rank and your rank would likely be lower. At a very basic level if you take two local business sites and one site does nothing other than have a great site and no links, yet the other has an equally good site but it has set up it's Google My Business and made sure it has links from relevant local directories plus maybe some local media/interest sites - Which do you think Google would trust more and rank higher? I have had some great sites that have come to me as clients - well optimised with nothing major needing changing to optimise the site but their rankings have all improved significantly in a short period of a few months through a structured outreach campaign earning links and ultimately increasing the DA. One final thought - even zero backlink experiments that have been successful have ended up naturally earning links and increasing their DA - to an extent worthy content in front of the right audience, such as social media, earns exposure/links.
Hope this makes sense...