I'm reminded of the xdcd "someone is wrong on the internet!" comment. I'm starting to get to the point where I don't feel like arguing or even explaining things; I just want to downvote, move on, and hope my moderation contribution has made the shape of the thread more useful to people.
If I had to explain my downvotes every time, I'd hardly ever bother, and I suspect that's true for most people. A forum where no one wants to take the time to moderate is not going to be a healthy place.