It would be better to ask whether someone, who is not a willing participant, is harmed by the activity. That's easier to establish.
The fact is, our collective idea of good and evil is internally inconsistent. It's still skewed by our repulsion to sexual deviancy and our fear of being judged by our peers. We really can't distinguish right from wrong. We aren't even able to apply the simple test you proposed.