If I didn't have a toddler to watch while I work much of the time, I'd probably just go all-in on VR (I'm currently working 30-50% through the Quest), but being able to see the world is kind of essential for those times haha.
As it goes with those kinds of things, you should definitely try the device out before making a conclusion, but given what you said earlier, I feel like you will like it.