Upvotes move posts up. Flags move posts down, and if you get a ton of them, it even makes the post disappear. That's all there is to it! What does it matter whether it's called a flag or a downvote?
As Matt himself says, it is a nuance that is lost on most.
I also draw no distinction between "spam" and "a link to someone's blog about how they were patted down at the airport." I would just as soon read about free Viagra and watches.