Even if "only" 10% of elite kids go on to become elite adults, 10% is orders of magnitude larger than the base percentage of adults who are elite athletes, musicians, etc. This doesn't sound "uncorrelated" to me so much as "not as strongly correlated as one might expect."
And describing something that happens 10% of the time as "rare" sounds a bit weird, like referring to left-handedness (also about 1 in 10) as rare.
e.g. If 1% of children are prodigies, prodigies are around 10x as likely to become elite as non-prodigies.
If 0.1% of children are prodigies, prodigies are around 100x as likely to become elite as non-prodigies.
Or in the rather unlikely case that 10% of children are prodigies, non-prodigies become elite at exactly the same rate as prodigies - 10%.