This feature could also come with the ability for 500+ karma to make such private comments visible (to make sure this privilege isn't being abused).
For example, in the following thread I would love to tell users why their comment does not constructively contribute to the discussion, but I also know that my comments on their comments don't contribute to the OP. There are many comments on there where a downvote is sufficient, but there are also borderline comments that merit an explanation as well. https://news.ycombinator.com/item?id=7469115