In terms of racism, its different but the same mechanism. Being compared to a minority race is not an insult (to most people). Its the fact, that racist people will use the word with vitriol. Racists and those they argue with will use the term in their arguments and gradually the use of the term will gain the conotation of a racist person. Hence, Negro -> Colored -> Person of Color -> <the next thing when PoC becomes racist>
Mongoloid, Retard, Special, individuals with learning disabilities are not entirely interchangeable (except as insults).