This is very time-consuming to do, however. Writing a detailed response is costly and repetitive. Pressing "downvote" is easy. You punish the commenter, but don't tell him/her why; it's up to them to figure it out on their own, which is sometimes difficult/impossible from their perspective.
An idea:
If you downvote, you have to pick a reason why you downvoted from various options (e.g. drop-down list, checkboxes). If you pick something, your downvote is cast. If you don't do anything, the downvote isn't cast.
In the options, you can list ~10-20 reasons to downvote (too mean, off topic, etc...). Select one or more items from the list, and submit it. What was selected appears in a "voting stats" page/section for said comment. Then the user can then get the gist of why they were downvoted, perhaps with a generic message saying how they could improve in the future.
The downside is that it might required a re-direct if you downvote (might be annoying on iPad/mobile devices), and it'll also take longer to downvote. I don't think the latter will be an issue since downvoting is rare, and I'd imagine people would be willing to fill out a quick web-form if they really think something deserves negative karma.