That would be awesome were it possible. Got any special insight into how to do that? I don't have a problem doing so, 16 yrs on reddit, if I ever downvoted even 30 times total, I'd be shocked. Others I've talked to make it seem like that's a daily or at best weekly total for them...
I think the trouble is that people fear others seeing the comment. Downvoting is used as a technique to prevent that contamination. It betrays some sort of mistrust in others' intellects.