Individuals judging you and making personal decisions on how to relate to you?
Or society making decisions to withdraw necessary services?
This is where firmly distinguishing between (individual freedom to associate / decide) and (social responsibility to deliver a necessary service) needs more clarity.
Should I be allowed to picket on public property in front of someone's house I disagree with? Or refuse to provide a service to them because I don't like them? IMHO, probably.
Should the city be allowed to turn off their electricity and water? IMHO, probably not.
Should VISA and Twitter be allowed to ban them? ... oof. That's a toughy.
Such responsibilities at some point seem an inherent consequence of running an economy where (1) companies are allowed to grow as big as they want & (2) "social" functions (i.e. services to all) are sometimes only provided by private parties (there is no government/public alternative).
* Leaving aside the protected-classes argument
I'm a socialist that thinks that companies should be transitioned into public ownership once they grow big enough.
It fixes the issue you describe in addition to the myriad of other issues associated with ever-increasing privatizion of what should be the commons.
And I really agree that the crux is "big enough" -- at some point, a phase change happens and regulation for social good needs to change too ('too big to be free'?).
We could pro/con public vs private-but-regulated management, but that's a dead horse and they're both valid options.