Part of the motivation also seems to be that we've fetishized disorders; having or claiming to have disorder X is a weird way to feel "special" or exceptional. Another part of it is that it functions as a kind of instrument for uptight people to gain a sense of exemption from social norms - real or merely perceived - that they feel burdened by or live in terror of "violating", but who are incapable of or unwilling to ignore them, because what would people say! "Oh, I'm weird, because I have disorder X. Can't help it, sorry! Can't judge me, because then you, not me, are the asshole!"
Needless to say, it's not very rational.
Which is not to say various disordered conditions don't exist.
One problem is that the language we've been given is one of defects (ADHD, ASD) and often people are stuck using that label even if they don't see their trait as a defect. So we're stuck saying "ADHD" because psychiatry decided that it's a defect. In reality, nearly every "disorder" has some other context-dependent benefit.
And while there's no "neurodivergence" in nature, there is a divergence from what is considered "normal" by society. "Normal" is an artificial construct, but we all have to live with it, and there are many aspects of my nervous system that clearly function in a way that's outside of two standard deviations from the mean, and this causes me a lot of problems in a society that can't tolerate that. These are very very real, and 95% of the time, I never say a word, and no one else has any idea that I'm suffering. I'm not telling anyone else that they're the asshole. Most neurodivergent people are living like this, despite what you're implying, that we're all just walking around being assholes to everyone we meet. That, is bullshit.