If any non-zero subset of reasonable people are so offended by a behavior that they'd leave the industry because of it, we have to cut it out.
So don't ask "would this bother me?" Ask "would it bother someone?" And since you can't predict this from inside your head, you have to rely on firsthand accounts of people being bothered. This seems like a good overview of such accounts.
People have jobs because they have to work, not because they would like to do it. If they leave the industry, it's because they don't find jobs in the industry worthy enough to endure. I'm sure more would stay for better rewards.
The reality is more that they are moderately to highly offended by a series of behaviors. It's usually not one single thing that someone runs into and says "that's it, I'm out", it's a long list of often smaller things that ultimately adds up to an intolerable experience.
> People have jobs because they have to work, not because they would like to do it.
Why can't we have both? Given that most of us spend the majority of our waking hours for the majority of our lives working, wouldn't we prefer to actually like what we're doing? Perhaps we'll never get there, but moving in that direction seems like a worthwhile goal.