Upvotes move posts up. Flags move posts down, and if you get a ton of them, it even makes the post disappear. That's all there is to it! What does it matter whether it's called a flag or a downvote?
As Matt himself says, it is a nuance that is lost on most.