To understand much of our language, Gnorts would have to already be aware that our words and symbols gain meaning from how they're used, and you couldn't, for instance, determine that a swastika is offensive (in the west) by its shape alone.
In this case, the term "colored people" gained racist connotations from its history of being used for discrimination and segregation - and avoiding it for that reason is the primary principle at play. There's also the secondary/less universal principle of preferring "person-first language".
But OPs point is broader: if you allow the bad people to just appropriate the symbol as their own, they're going to gradually take over everything. Never mind swastikas; we're at the point where making an okay sign can be misconstrued as a white nationalist gesture, and people self-censor themselves accordingly.
There's also the reverse problem here, where, if you tie such things so strongly to symbols in popular opinion, then loud condemnation of such symbols is used to "prove" that one is not a bad person. For a major ongoing example of this look at Russia with its cult of "we defeated the Nazis therefore we're definitely the good guys".
At the end of the day, it's really just a lazy shortcut. The bad people are bad because of their ideas and actions, not because of their symbols. If we always look at the ideas and actions, the symbols are irrelevant, and we don't have to surrender them to the bad guys' claims.