To understand much of our language, Gnorts would have to already be aware that our words and symbols gain meaning from how they're used, and you couldn't, for instance, determine that a swastika is offensive (in the west) by its shape alone.
In this case, the term "colored people" gained racist connotations from its history of being used for discrimination and segregation - and avoiding it for that reason is the primary principle at play. There's also the secondary/less universal principle of preferring "person-first language".
Switching to another word isn't ceding power to the bad people. It's taking away their power to redefine things. It's letting them have the now-useless word exclusively, which will become associated with their speech, and not the original meaning. The original meaning is reclaimed by using a new not-yet-soiled word for it, and the cycle continues.