I tried switching to a feature phone and was surprised how often I use a smartphone; and how many people, banks, government organizations, restaurants, etc, assume that you have one.
There are vanishingly small numbers of people who will insist on a perfect-or-nothing approach to smartphones. This market segment is unserviceable. Sure, the size will be right, but it won't have the right battery size, or the battery has to be swappable on-the-go, or it didn't have quite the right sd card option, or maybe the software isn't 'polished' enough, or it had to have two headphone jacks. There will be something 'not good enough' and therefore it's passed over even though they want a 'small' phone.
There is a smartphone that meets the needs of a small-phone purchaser, after all, if small is the requirement - the iphone mini. But purchasing that would require some compromise on your hardline requirements, which will be different to someone else's hardline requirements such as a swappable battery.