If any non-zero subset of reasonable people are so offended by a behavior that they'd leave the industry because of it, we have to cut it out.
So don't ask "would this bother me?" Ask "would it bother someone?" And since you can't predict this from inside your head, you have to rely on firsthand accounts of people being bothered. This seems like a good overview of such accounts.
(Note, however, that the contrapositive is "if you're doing something important, you'll make someone angry", not "if you're making people angry, you're doing something important"!)