There's a substantial difference between downvotes on comments and downvotes on stories. Upvoting or downvoting a story is a vote for whether you agree or disagree that the linked story contributes to "hacker news". Upvoting or downvoting a comment
should be a vote for whether or not the comment constributes substance to the conversation.
I'd be in favor of reversing what can be down-voted: allow submissions to be downvoted, but only allow comments to be upvoted. Then the comment 'vote' more easily becomes "is this good?" (or at least removes the "is this something you disagree with?") and we can maintain a 'flag' on comments for the trolls/spam.