I agree the potential for damage is much higher with downvotes, but there's an asymmetry there.
The trick is to make downvotes cost a little bit of karma, IMO. And while I'm on the topic, upvotes shouldn't be as free as they are, either.