I think it really conflicts with the "optimise for curiosity" aspect of the site. Downvoting comments for disagreement does not help curiosity, as it just downranks insightful yet unpopular opinions, which obviously does not help encourage curiosity. The real offtopic/flamebait comments get flagged anyway, so downvotes aren't needed to keep the site functional and interesting.
(last sentence was slightly edited for clarity)