I was about to say you can't flag comments so downvotes are the only option, but I realized I have only been assuming that for years and never really questioned it. Looks like if you click on the "n [minutes|hours|days] ago" link on a comment, there is a "flag" option there. Now I know. So the questions is, why even have a downvote button?