Those of us accusing and talking about it have no power -- thus there is literally no harm, and possible good in, putting them on the defense about this.
edit: In fact, the First Amendment of the Constitution essentially directly upholds the idea of "people saying whatever they want" in this regard.