That phrasing (in particular "does" vs. "only") makes it sound like being bothered by the first thing is inherently justified but being bothered by the second thing isn't. And why do you specify the condition that will cause the second thing to bother the people but leave it ambiguous for the first thing?
I don’t think describing the conditional thing as conditional implies any judgement.
Fixed it for you.