Like the OP, I switched from Android (Pixel 3a) to an iPhone SE 3 specifically for the smaller form factor. After using it for over a year, I've found the trade-offs in battery life and camera quality are too significant for my daily use.
These limitations aren't an issue when I'm at home or my desk with easy access to a charger. However, they become acute the moment I'm out for the day. For example, using GPS for navigation or connecting Bluetooth accessories becomes a liability. I can't rely on the phone to last. Also, photos are noticeably more pixelated, and the quality drop-off is clear compared to larger, contemporary phones.
This thread is evidence that the niche for small phones exists. But it's for people willing to accept these compromises by carrying a dedicated camera, a power bank, and using wired peripherals. For me and as the market suggests for most consumers, small phones just doesn't work out as reliable all-in-one devices. I'll probably wait till early next year to pick up one of the new iPhones after they iron out the initial kinks.