Even if "only" 10% of elite kids go on to become elite adults, 10% is orders of magnitude larger than the base percentage of adults who are elite athletes, musicians, etc. This doesn't sound "uncorrelated" to me so much as "not as strongly correlated as one might expect."
And describing something that happens 10% of the time as "rare" sounds a bit weird, like referring to left-handedness (also about 1 in 10) as rare.
So I think human brain development is like some kind of optimization algorithm, like simulated annealing or gradient descent. I think this because there is way more complexity in the brain than there is in human DNA, which has pretty low information by comparison. Anyway, child prodigies occur when the algorithm happens to find a good minimum early on.