If any non-zero subset of reasonable people are so offended by a behavior that they'd leave the industry because of it, we have to cut it out.
So don't ask "would this bother me?" Ask "would it bother someone?" And since you can't predict this from inside your head, you have to rely on firsthand accounts of people being bothered. This seems like a good overview of such accounts.
But reasonable people were offended by Brandon Eich donating to an anti-gay marriage proposition, and other reasonable people were offended that he got ousted from Mozilla.
Some people are offended by Damore and been quite vicious towards him, others are saying he's right about things (including you pretty much can't be a conservative in Silicon Valley).
Isn't reality more complex than this idea? Its easy enough to say "don't comment on a coworkers physical appearance" or something equally stupid, but "any non-zero subset" offended by any particular behavior doesn't seem reasonable.