Yeah I'm in the boat of "people shouldn't use downvotes for differing opinions", but I do downvote things I disagree with when I think they are "in bad faith". This is definitely a vague and subjective metric, but includes things like the demonstrable falsehood you mentioned, as well as stuff like continuing to make the same point while seemingly failing to read or incorporate what their interlocutor is saying.
I guess what I'd say is that most disagreements are honest ones - people coming to different conclusions on subjective things - and that's the kind of disagreement I don't think downvotes should be used for.