But it's not about material harm, it's about boundaries. And really about which boundaries can be set. For example these days we realize being married doesn't mean the other person can force you to have sex. But that wasn't obvious at a certain point in history! The boundary couldn't be realistically set because it wasn't supported by legislation. You can't set boundaries without power.
Let's consider an example. My premise here is that boundaries depending on harm done is insufficient to motivate existing legislation-supported boundaries that basically everyone would agree with.
Imagine someone that gives everyone hugs. They are gentle, mostly. They particularly like giving you hugs, because they know you don't like it. No matter what you say, they won't stop. You can't get them to leave you alone, and your work requires you to be in that office. Actually they have access to all offices of business with open positions in your field. They even show up at and in your house. They just follow you until you get tired. You can't change your locks because the company that services your house only supports that lock. You can't get them fired. Most of your coworkers don't care that much, and some like it. A few people really don't like it and have sophisticated ways to track him so they can avoid people like him most of the time, but they spent a lot of time on boats to do that and no one will hire them. You can't do that because you are neither technical enough nor willing to forego showers and employment. If you retaliate or lash out you'll be arrested. You are complaining about it, suggesting someone makes it illegal to gratuitously touch someone who doesn't want to be touched, and someone asks you "What is the harm? I know you don't like it, but how are you being harmed?"
How do you answer?