Flag comments (essentially public shaming). Look at how Quora does it. If a user's reply is not in line with the question, other users flag it as "not helpful" (and explain why below).
So the more the people who flag stupid comments (instead of just downvoting), the more these comments should descend to the bottom, regardless of number of votes.