I draw the line at spaces, usually, because functionally there are usually spaces around where I care to have alignment.
But also, since I'm writing a character stream, I care about being able to see it as a monotonic stream of distinct characters (which is why I hate hate hate ligatures for e.g. "!=") and moving characters a tiny bit doesn't break that.