But OPs point is broader: if you allow the bad people to just appropriate the symbol as their own, they're going to gradually take over everything. Never mind swastikas; we're at the point where making an okay sign can be misconstrued as a white nationalist gesture, and people self-censor themselves accordingly.
There's also the reverse problem here, where, if you tie such things so strongly to symbols in popular opinion, then loud condemnation of such symbols is used to "prove" that one is not a bad person. For a major ongoing example of this look at Russia with its cult of "we defeated the Nazis therefore we're definitely the good guys".
At the end of the day, it's really just a lazy shortcut. The bad people are bad because of their ideas and actions, not because of their symbols. If we always look at the ideas and actions, the symbols are irrelevant, and we don't have to surrender them to the bad guys' claims.