General population doesn't buy phones every year and they don't want a nerfed phone when they have to pay 500-1000 $/€s. So they gravitate towards higher end ones.
Companies including Apple has always treated the small size as an entry to mid segment phone. The only exception I know is Sony z3 and z5 compact which suffered heat and battery swelling issues due to Qualcomm messing 810 series SoCs up.
Companies also want you to buy the most expensive phone. So they market the premium models and train their store personnel to sell more of the premium line. If they stop intentionally nerfing the smaller phones, I think there is a market there. However, it will still be smaller.