Also, downvoting past 1 karma in general is a bad thing. It adds bias to the comment for future readers.
It really seems like downvoting in its current form is fine, and if any changes need to be made then capping downvoting at 1 karma might be a good idea.
I also don't see why you need to reward the community. The right to down vote is not different from the right to up vote, and any user should have it. Up vote is simply "I want more of this", and down vote "I want less of this".
The same machinery used for up-votes of bad links can be applied to down-votes of good links, so you can protect the site from general stupidity.
Therefore, if the community has deemed a comment as stupid (karma < 1), any future readers will read the comment through tinted glasses. But wait a minute, the community really hasn't said that. It was only one or two users that did.
Let's reexamine the purpose of karma on a comment. The purpose is not simply to punish the poster of a comment if it gets downvoted. It's to move the noise down to the bottom of the thread, where it can be ignored.
So I propose an alternative: Let silly comments stay at the bottom with one karma point, and encourage users to upvote more interesting comments higher. This way, users won't be biased against any comments (They won't know if a comment has been downvoted because it might simply be that no one has upvoted it) and the noise can still stay at the bottom.