FWIW, I try to use "male/males" to refer to sex, and "man/men" to refer to gender, since AFAICT, there are no terms that clearly refer to gender.
"men" and "women" should not have their meaning diluted. They already have the meaning of "male" and "female".