Well, I think that unlike women or bin-binary people, a man wouldn't be able of accusing another man of sexism in his direction.
Regardless, we’re talking about social judgements. In a legal setting the burden of proof would be on the accuser.
And like many legal outcomes, just being accused is its own stigma. Someone accused of murder but then later (let's say objectively, truthfully, correctly) found not guilty will expect to face social discrimination and alienation. It's not right, but it's unfortunately how people operate.