If so, it seems as though account-level signal/noise weighting could help. New accounts and ones that are consistently downvoted could be given less prominence in the UI (until upvoted, of course).
The idea is similar to the current behavior of requiring a minimum karma count before allowing users to flag/downvote.