The best Android phones I've had were Nexus 5 and Nexus 5X. I thought the 5X were a little bit bigger than I prefer, but over all a great device. After that I went through a bunch of different phones, first a cheapo Motorola at 5.7" (smallest I could find without complete garbage specs) and then a Samsung A10 at 6". They were both impractically big, despite having quite large hands.
The only way I could find to have a smaller screen was to switch over to iOS. Currently I have an iPhone SE (iPhone 8) at 4.7" which is lovely, but I would have preferred to stay with Android as it's the only Apple device I currently own.