I agree with you but I think this idea of being "fair" is something that is said but no-one actually believes in. Most recent government is one of the most extreme examples of this: do things that annoys everyone, say you are just being "fair" because everyone is annoyed...it doesn't make sense.
To say this another way, there is genuinely an easier option: stop doing things that people do not want.