It's something I've suggested before, getting rid of downvotes and replacing them with flags.
Really, you're stuck with the same problem: just as there could be mass upvotes of "poor" comments and mass downvotes of "good" comments, there could be mass flagging of "good" comments as "spam", "dumb", or "mean".