Even if "only" 10% of elite kids go on to become elite adults, 10% is orders of magnitude larger than the base percentage of adults who are elite athletes, musicians, etc. This doesn't sound "uncorrelated" to me so much as "not as strongly correlated as one might expect."
And describing something that happens 10% of the time as "rare" sounds a bit weird, like referring to left-handedness (also about 1 in 10) as rare.
https://www.simplypsychology.org/base-rate-fallacy.html
> For example, given a choice of the two categories, people might categorize a woman as a politician rather than a banker if they heard that she enjoyed social activism at school—even if they knew that she was drawn from a population consisting of 90% bankers and 10% politicians (APA).
The general population is much larger than the population of child prodigies.