Such labels are a part and parcel of in-group/out-group psychology. In 2019, by analogy, we should all be wary when people are starting to probe those "ports." I'm pretty sure that I've been seen as both by different people at different times, though recently on HN I've been spending a lot of energy defending a particular principle. Going way back, I'd been pre-judged as a woo-woo hippy, a yuppie, and all sorts of different things since.
If we take a step back and look at the broader arc of history, what we have to be wary of are narratives which attempt to paint entire groups of people as morally inferior threats. (For a variety of reasons, including world view.) These narratives always precede the great evils of history. These narratives always deserve great skepticism -- we should always "follow the money" and see who benefits from the pushing of the narrative. We should ask who stands to gain power.