Like the OP, I switched from Android (Pixel 3a) to an iPhone SE 3 specifically for the smaller form factor. After using it for over a year, I've found the trade-offs in battery life and camera quality are too significant for my daily use.
These limitations aren't an issue when I'm at home or my desk with easy access to a charger. However, they become acute the moment I'm out for the day. For example, using GPS for navigation or connecting Bluetooth accessories becomes a liability. I can't rely on the phone to last. Also, photos are noticeably more pixelated, and the quality drop-off is clear compared to larger, contemporary phones.
This thread is evidence that the niche for small phones exists. But it's for people willing to accept these compromises by carrying a dedicated camera, a power bank, and using wired peripherals. For me and as the market suggests for most consumers, small phones just doesn't work out as reliable all-in-one devices. I'll probably wait till early next year to pick up one of the new iPhones after they iron out the initial kinks.
(After all, easiest way to increase battery size is to increase the smallest dimension. Add 1mm to a 4-4.5mm thick battery and you'll increase the battery size by 22-25%. Make the iPhone 13 Mini as thick as its camera bump and you would probably add ≈2.4mm, which would make the battery 60% larger)