I don't think we should make it any more complicated. You might benefit (on the margin) from some feedback, but that makes the conversation more noisy for everyone.
I can say that personally, if I downvote something without also replying to it, it's either because I don't have time to write a helpful response, or because I know my own contribution would make things worse.