The eyes are adapted to see daylight levels of light during daytime. If your ambient lighting and screen don't match this, then you're not using your body in the way it's evolved to function. What we need more of are screens with anti-glare and a wide range of brightness levels.
What might have a worse impact on your eyesight is the effects of Myopia, which is being exacerbated by basically every portable device with a screen. Anyone who has the technological literacy to enable dark mode (and advocate for it) is likely someone who gets above-average screen time and is therefore someone at a higher risk of myopia.