"In practical terms, downvotes are so extraordinarily rare on HN that they almost don’t exist."
It's not because you don't see them directly a lot that they don't have an impact. A comment that ends at +3 with 7 upvotes and 3 downvotes would have ended at +7 (or more because of the "momentum effect") if there weren't downvotes.