In a hardcore surveillance society, almost any innocent act can be harmful to one. If government, for anti-terrorism reasons, has a rather complete picture of our lives, then we need strong safeguards against how they can act based on that picture.
And the same goes for the private sector, which might form that picture for business reasons.