Putting people at ease around you (especially customers) is a critical entrepreneurial skill.
You can't blame people for being cautious when a lot of people are buying into victim-narratives and convinced to act against their "oppressors".
And even if one were to believe that that is unlikely too, nothing guarantees that someone else won't think it sexist. For example, I remember some panel with four scientists (three men and one woman) that was discussed in HN a while ago; at some point, someone in the audience (I think she was a journalist) yelled at the moderator to "let her speak"... Even though the scientist herself didn't think the moderator was doing anything wrong.