You should have to enter an explanation when you downmod something. Then the recipients won't be left confused, and it will enforce responsible use of the privilege. I feel this is a better solution to the abuse problem than only allowing downmods for 24 hours.
Alternatively, cap downvoting at 1 karma, so that comments can't go below that threshold.
(Repeating from my previous post): Letting karma go below 1 adds bias to the comment for future readers, but that bias doesn't reflect how they might feel.