The thing is socialization: women are taught from a shockingly young age to be wary of men, dark alleys, tunnels, parks, everything. Black people to be careful around police.
Of course even if you do not belong to a group of people who face regular discrimination, you can still be mindful of the issues the groups face - but it won't ever be an as natural part of your thinking.