Another thing I've considered is having specific types of flags on comments, and having them have different effects. E.g. there could be a flag for incivility, and if you got enough of those (maybe in proportion to your total number of comments) you'd actually get kicked off the site temporarily.
alex is right that automatically escalating based on #flags is a blunt instrument. You'd need to perform the escalation manually, you'd need multiple people doing it, and you'd need the decision-maker to attach their name to it ("kn0thing marked this uncivil") so the watchers can be watched in a lightweight manner.