The only people who like small form factor phones are tall men.
Every other demographic still needs two hands for a 5" phone and thus opts for more screen real-estate almost all of the time.
Sorry to sound snarky, but make it smaller then. 4" sounds good to me.