That's the same argument that you see on Reddit: "If only people would downvote things on /r/[x]/new they didn't want to see". But obviously that's not making much of a difference. There's just not enough voting and submission activity for the subset of users that really care to influence things that much.