Also, downvoting past 1 karma in general is a bad thing. It adds bias to the comment for future readers.
It really seems like downvoting in its current form is fine, and if any changes need to be made then capping downvoting at 1 karma might be a good idea.
I also don't see why you need to reward the community. The right to down vote is not different from the right to up vote, and any user should have it. Up vote is simply "I want more of this", and down vote "I want less of this".
The same machinery used for up-votes of bad links can be applied to down-votes of good links, so you can protect the site from general stupidity.