https://static.googleusercontent.com/media/research.google.c... and https://norvig.com/chomsky.html
In short, Norvig concludes there are several conceptual approaches to ML/AI/Stats/Scientific analysis. One is "top down": teach the system some high level principles that correspond to known general concepts, and the other is "bottom up": determine the structure from the data itself and use that to generate general concepts. He observes that while the former is attractive to many, the latter has continuously produced more and better results with less effort.
I've seen this play out over and over. I've concluded that Norvig is right: empirically based probabilistic models are a cheaper, faster way to answer important engineering and scientific problems, even if they are possibly less satisfying intellectually. Cheap approximations are often far better than hard to find analytic solutions.
It's true that it's less satisfying and less attractive, but these subjective adjectives are based on relevant objective truths, namely that LLMs are difficult or impossible to analyze from the outside, and at a coarse level they're the knowledge equivalent of pathological functions. Calling them "intelligent" is to privilege a very limited definition of the word, while ignoring all of the other things that we normally associate with it.
I don't want us to make an AGI or anything like it for both humanist and economic reasons, but if we make one, I think it's very likely that it has to have more internal structure than do LLMs, even if we do not explicitly force a given structure to be there.
(I am not an expert.)