Flip it around, don't accept the questioners' premise that you need to defend your position. Challenge them to explain why they think a third party should be granted access to your info. And be ready to tell people 'no thanks' and that you are not willing to explain why. Resist taking a subservient position when you have what they other party wants, make them justify the need (this is obviously less practical advice for gov't surveillance than for corporate).
I do this frequently and while it can be a bit awkward when dealing with marketing or PR types, as long as you are polite about it things work out. And anyone pestering you with repeated requests for data or an explanation can receive a less polite response.