Allow every user to have downvote suggestions. Allowing people to suggest that a comment should be downvoted allows those with sufficient privilege to hone in on the bad comments and it lets other users be more involved. Possibly, track a user's 'discernment' level - increase it when a suggestion is acted upon. Use this to weight how much that user's suggestion effects a comment's "downvote-suggestion rank" as it is shown to the trusted users. Promote users to trusted status when their discernment reaches a certain point. This discernment level would both measure a user's interest in maintaining the site as well as predict how good they would be at it.
Maybe even do this in general (for up and down votes): all users cast only suggestion votes. Trusted users cast the real votes.