That observation has echoes of the music industry - another extremely top-heavy creator business. There are formulaic ways to make "good enough" and "catchy enough" songs, but the window for "X enough" keeps shifting. Cranking out grunge won't be sustainable in the age of K-pop.
But the massive runaway hits have been predominantly outliers for their age. They have veered far enough from the mainstream to be interesting in new ways, different enough, and surprising enough to break through.
But to predict in advance what kinds of outliers will win the lottery? Largely random, indeed.
Dear God, I've looked into his discography[1] and nearly every album I think of as great from the last 30 years is there. Seasons in the Abyss, The Life of Pablo, 99 Problems, SOAD self-titled + Toxicity, The Geto Boys self-titled, Licensed to Ill... Is this man a hit printer or something? Really shows that Metallica went to him with Death Magnetic after the joke called St. Anger lol
[1] https://en.wikipedia.org/wiki/Rick_Rubin_production_discogra...