Right, so the whole premise of your indifference or opposition to the privacy argument is that people should not have prejudices or (wrong) assumptions. Isn't that too idealistic and to rid people of the prejudices and figure out right moral standard for behaviour - will it not take many more generations, if at all it happens? Till then; till we figure out the right _prejudices_; till all of humanity naturally elevates to the right moral standard, shouldn't we be wary of those bad agents who can abuse others by breaking into their private matters?
Your premise, in short, assumes an ideal world where none is troubling others for their private acts, which unfortunately isn't the case yet.