> I've often been downvoted below zero within a minute of posting
Quick downvotes typically do more good than harm. In my experience both voting and receiving votes, a undeservedly grey comment gets more upvotes than a comment that hasn't been downvoted.