Upvoting is just as censoring as downvoting is, which is to say, it's not censorship at all. "Free speech" does not mean that all speech is equally valuable. Some speech is worthless noise, and recognizing that is the only way to have useful discourse in an open forum.