1. Bigger chassis allows a bigger battery, longer up-time. This is probably the biggest issue.
2. Having gotten used to a 6+" for a while, the small ones appear almost too small. Books are on average bigger than that for a good reason.
3. Very naively speaking, bigger looks better and thus more expensive. Users are therefore more willing to spend more money, whereas a smaller phone will look disproportionately expensive and, as per the first point, now less powerful.
Sony was the last one other than apple to try that, as far as I can tell.
However, cheaper smartphones still exist in that space, they just don't come with newer hardware.