That's true, under the existing system. And that makes it attractive to reform the system, to make it easier for third parties to grow.
For example, approval voting was passed in Fargo recently: https://www.electionscience.org/commentary-analysis/approval...
tl;dr -- third parties can't grow as long as "splitting the vote" is a thing. But some voting systems have that and others don't.