I wonder what the userDB says about any scarcity of upvoters? Are there many others like me? Am I not pulling my weight? Should I do it more?
If the ratio is too low (eg: spam, shallow content, same old tired point), it's not an issue to downvote IMO. That helps with the current post and also encourages good behaviour in the future.
This is not the same as downvoting with a point that you personally disagree with. This is censorship and toxic to the community.