I'm concerned with hypocrisy and fanaticism. I've seen what a tyranny looks like: a mandatory mindset with strong social pressure to conform, free thinking as a public sin and people becoming hypocrites, the ones not dumb enough to be fanatics.
It sounds like you want to distance consequentialism from tyranny. As a consequentialist, I can see the draw, but I think that it's vital that we admit that consequentialism, improperly applied, can be used to justify tyranny. If we pretend that can't happen, then we won't "notice the skulls"[1]--that is, we won't notice when consequentialists are supporting tyranny in our midst, because we've been arguing all along that that can't happen. It's vital that we see where consequentialism can go wrong so that we can take steps to prevent it from going wrong.
[1] https://slatestarcodex.com/2017/04/07/yes-we-have-noticed-th...