The eyes are adapted to see daylight levels of light during daytime. If your ambient lighting and screen don't match this, then you're not using your body in the way it's evolved to function. What we need more of are screens with anti-glare and a wide range of brightness levels.
What might have a worse impact on your eyesight is the effects of Myopia, which is being exacerbated by basically every portable device with a screen. Anyone who has the technological literacy to enable dark mode (and advocate for it) is likely someone who gets above-average screen time and is therefore someone at a higher risk of myopia.
Well, I'll meet the dark-mode people halfway, and say that it should always be available to users who want it.
No one is saying that a personal preference for dark mode is invalid, just that it shouldn't be the default option, or worse, the only option.