It sometimes breaks out of the grid, but there still is a grid. Things are sometimes not micro-aligned but still macro-aligned, and that matters (to me).
But also, since I'm writing a character stream, I care about being able to see it as a monotonic stream of distinct characters (which is why I hate hate hate ligatures for e.g. "!=") and moving characters a tiny bit doesn't break that.