One problem is that the language we've been given is one of defects (ADHD, ASD) and often people are stuck using that label even if they don't see their trait as a defect. So we're stuck saying "ADHD" because psychiatry decided that it's a defect. In reality, nearly every "disorder" has some other context-dependent benefit.
And while there's no "neurodivergence" in nature, there is a divergence from what is considered "normal" by society. "Normal" is an artificial construct, but we all have to live with it, and there are many aspects of my nervous system that clearly function in a way that's outside of two standard deviations from the mean, and this causes me a lot of problems in a society that can't tolerate that. These are very very real, and 95% of the time, I never say a word, and no one else has any idea that I'm suffering. I'm not telling anyone else that they're the asshole. Most neurodivergent people are living like this, despite what you're implying, that we're all just walking around being assholes to everyone we meet. That, is bullshit.