I do not have these sorts of worries around people with whom I am well acquainted. I know I'm not prejudice and I trust my friends to interpret my words and actions in good faith. But I recognize there is a minority of people out there who will intentionally take what I say or do in bad faith if it benefits them. People like that exist in all groups, regardless of physical traits. When interacting with a minority or protected class, I have to worry just as much about interpretations of those around me as that of the person I'm interacting with. In a world where accusations of prejudice are often met with a guilty-until-proven-innocent mentality, any such accusation is very damaging regardless of who it comes from.