As to flagging, I think a bigger problem is activism-upvoting, in other words upvoting so others see a story and not because it's interesting.
If you don't trust the average user with flagging powers because of how they affect the front page, why do you trust them to upvote good articles onto the front page? Clearly only select users should be able to upvote articles.