Some cultures have known that things are not binary for a very long time:
Being "a wife to start a family" requires a person with female sex, and is typically associated with female gender. But that association is not required, and has not been so across all human cultures and all time.
FWIW, I try to use "male/males" to refer to sex, and "man/men" to refer to gender, since AFAICT, there are no terms that clearly refer to gender.
"men" and "women" should not have their meaning diluted. They already have the meaning of "male" and "female".