What you're describing has always seemed to me as an excellent example why women-only spaces are not necessarily about people-who-identify-as-women, but about people-who-others-identify-as-women. Other people treat you a certain way because of how they perceive you, and you want to have a place to talk about that with other people treated the same way. Does that sound at all correct?