Personally I think downvotes should also be calibrated by user, a person who rarely downvotes should carry more weight in a downvote than someone who downvotes everything, and perhaps even have user-to-user calibration where your downvotes against someone become weaker the more you target them. It would help nerf people who scour a post history looking for more comments to downvotes, and help prevent downvote gangs of people who consistently downvote the same person no matter what they post.
A certain amount of dissenting opinion is necessary to keep discussions novel and break up echo chambers, but you don’t want to allow so much that you become poisoned by it.