The eyes are adapted to see daylight levels of light during daytime. If your ambient lighting and screen don't match this, then you're not using your body in the way it's evolved to function. What we need more of are screens with anti-glare and a wide range of brightness levels.
What might have a worse impact on your eyesight is the effects of Myopia, which is being exacerbated by basically every portable device with a screen. Anyone who has the technological literacy to enable dark mode (and advocate for it) is likely someone who gets above-average screen time and is therefore someone at a higher risk of myopia.
Depends on if you target a dark or bright ui. If you target a bright environment then you want to work in a bright environment and vice versa, otherwise you bias the work for the wrong environment as you say.