But, what if there was a person in your group that was actually offended? And couldn't say so because they'd be out of the group? Does it matter? (To me, yes). Does it matter enough to change the group's behavior? (Maybe). Does it matter enough to change the entire world's behavior?
It's through these that we may move away from disproportionate responses to what feels uncomfortable and find common understanding with each other, but alas, never perfectly.