But the weight of every flag should depend on how well user's downvoting correlate with your own downvoting.
If there is no correlation between user and you - then downvoting should not affect ranking.
That way you would get types of stories you don't like on HN to be quickly downvoted.