These are two completely different meanings of the word though? Are you arguing that they need to carry the same semantics?
Should we not call smart people "bright", because people might think light bulbs are intelligent too?
Also, if I call someone fat, that might be demeaning. But it also might just be a description.
Of course nobody wants a negative description applied to them, and it might make them sad, but it's not because of the word we're using: it's because we're ascribing that description to them.