It's like wondering how well your shoes fit your feet, forgetting that shoes are made and chosen to fit your feet in the first place.
Chomsky also talks about these kind of things in detail in Hauser, Chomsky and Fitch (2002) where they describe them as "third factors" in language acquisition.