Now, sex has many correlating genetic and phenotypic factors, and so if someone possesses those characteristics but is simply infertile, then we would still be correct in identifying them as being of a particular sex.
I think in the context of intersex people (at least those who are infertile) that the idea of "assigning" sex makes sense, because there is some natural ambiguity there. But just because determining a person's sex is arbitrary in some circumstances does not mean it is arbitrary in all circumstances.