> so if process described in 1) is going to lead towards a working general intelligence, there's a good chance it'll stumble on the same architecture evolution did.
I wouldn't be so sure of that. Consider that a biased random walk using agents is highly dependent on the environment (including other agents). Perhaps a way to convey my objection here is to suggest that there can be a great many paths through the gradient landscape and a great many local minima. We certainly see examples of convergent evolution in the natural environment, but distinct solutions to the same problem are also common.
For example you can't go fiddling with certain low level foundational stuff like the nature of DNA itself once there's a significant structure sitting on top of it. Yet there are very obviously a great many other possibilities in that space. We can synthesize some amino acids with very interesting properties in the lab but continued evolution of existing lifeforms isn't about to stumble upon them.
> the symbolic approach to modeling the world is fundamentally misguided.
It's likely I'm simply ignorant of your reasoning here, but how did you arrive at this conclusion? Why are you certain that symbolic modeling (of some sort, some subset thereof, etc) isn't what ML models are approximating?
> the meaning of words and concepts is not an intrinsic property, but is derived entirely from relationships between concepts.
Possibly I'm not understanding you here. Supposing that certain meanings were intrinsic properties, would the relationships between those concepts not also carry meaning? Can't intrinsic things also be used as building blocks? And why would we expect an ML model to be incapable of learning both of those things? Why should encoding semantics though spatial adjacency be mutually exclusive with the processing of intrinsic concepts? (Hopefully I'm not betraying some sort of great ignorance here.)
I'm not the poster, but my answer would be because symbolic manipulation is way too expensive. Parallelizing it helps, but long dependency chains are inherent to formal logic. And if a long chain is required, it will always be under attack by a cheaper approximation that only gets 90% of the cases right—so such chains are always going to be brittle.
(Separately, I think that the evidence against humans using symbolic manipulation in everyday life, and the evidence for error-prone but efficient approximations and sloppy methods, is mounting and already overwhelming. But that's probably a controversial take, and the above argument doesn't depend on it.)
Even if we were to obtain evidence clearly demonstrating that LLMs don't reason symbolically, why should we interpret that as an indication of what humans do? Certainly it would be highly suggestive, but "hey we've demonstrated that thing can be done this way" doesn't necessarily mean that thing _is_ being done that way.
They make people willing to seriously consider a wider ranger of possibilities. Without the example of LLMs, people tended to be very attached to a "hoomanz special, you need to have the exact same physical substrate to do anything remotely similar." With LLMs, now you have the equally misguided (IMHO) "LLMs talk like people, so they must be doing the same thing as people".
> Couldn't you have argued the same thing prior to LLMs? That evolution is a greedy optimizer etc etc therefore humans don't perform symbolic reasoning.
Could and did. I've long argued that usually when we think we (or others) are thinking logically, that that's a retconned explanation for a decision or behavior that was really arrived at in a messier and more error-prone but also more powerful mechanism. ("Powerful" as in, with wider applicability and generalizability. Not necessarily more capable of arriving at "correct" solutions.)
> humans don't perform symbolic reasoning [is] merely a hypothesis - there's zero evidence one way or the other
"Zero evidence" is inaccurate. There is lots of evidence for what Kahneman calls system 1 and system 2 thinking. (The reports of the death of this theoretical model are greatly exaggerated -- while lots of the research covered in "Thinking, Fast and Slow" has been debunked, the existence of and distinction between these modes of thought are empirically supported.) There's also tons of evidence for how we graft explanations for our past decisions onto them after the fact (even when we are fooled into thinking we made a decision that we didn't, we'll still generate and believe a logical explanation).
But also, I'm not claiming that neither LLMs nor humans ever reason symbolically. I think both do, occasionally. I claim that the bulk of the behavior of both LLMs and humans is not decided upon via symbolic reasoning. The basic reasons are similar -- it's cheaper and more efficient to use other approximate mechanisms, and both of us learn to do what works rather than what is correct, at least most of the time.
> Even if we were to obtain evidence clearly demonstrating that LLMs don't reason symbolically, why should we interpret that as an indication of what humans do? Certainly it would be highly suggestive, but "hey we've demonstrated that thing can be done this way" doesn't necessarily mean that thing _is_ being done that way.
Agreed. But "highly suggestive" is all I'm going for. (And only highly suggestive that neither of us rely heavily on symbolic reasoning, not highly suggestive that we work the same way as LLMs.)
It's tricky, because LLMs are almost designed to introduce as many confounding factors as possible. For example, it's popular to claim that you can't have "real" intelligence without embodiment. (Though that position seems to be declining in popularity with LLM advances.) You need skin that can feel a breeze, a body that can feel pain, a mind that can suffer. You need neurons that live in a chemical bath whose composition and history are part of the processing mechanism. But LLMs are trained out tons and tons of output that was generated by embodied creatures, and so that input data "carries along" the results of a processing mechanism that relies on embodiment. An LLM that claims to enjoy long walks on the beach and the feeling of sand between its toes, and that gets grumpy during the dark season, isn't lying. It was built to emulate the output of beings that do enjoy sandy toes, and can generate new "thoughts"/outputs that are produced via mechanisms that take that into account.
To the extent that chatbots live, they live vicariously through us.
> Possibly I'm not understanding you here. Supposing that certain meanings were intrinsic properties, would the relationships between those concepts not also carry meaning? Can't intrinsic things also be used as building blocks? And why would we expect an ML model to be incapable of learning both of those things? Why should encoding semantics though spatial adjacency be mutually exclusive with the processing of intrinsic concepts? (Hopefully I'm not betraying some sort of great ignorance here.)
I probably shouldn't respond to this part, because I don't really agree with the original assertion. Or rather, I think this ends up boiling down to a disagreement over semantics, and so isn't a particularly interesting question.
Relationships between concepts covers a lot of what "meaning" is. You can teach a computer to translate from language X to Y purely based on it learning the relationships of words to each other in each language, and then generating a mapping between the weight-graph of X to the weight-graph of Y. (Yeah, citation needed; I remember reading some specific evidence for this, but I don't remember where.) So you can get a long way with just relationships.
At the same time, I don't think that proves that the relationships between concepts are everything. A human getting burned and learning the word "hot" could be described as "hot" having an intrinsic meaning. But you could equally describe it as a relationship between the action taken, the sensation experienced, and the phonemes heard. If all those are "concepts", then the relationships between concepts are everything. If they're not, then you can call something intrinsic. Personally, that strikes me as a pointless philosophical question.
I guess you could argue that if you have an LLM trained on mostly English but also enough Chinese to be able to translate, and it generates text including the word "hot", then if you compare that to the same LLM generating text including the Chinese word for hot, that there's more opportunity for drift in the Chinese output. The first case has the chain of a human feeling pain => writing text containing "hot" => generating text containing "hot", whereas the second has the chain of a human feeling pain => writing text containing "hot" => encoded associations between English and Chinese concepts embedded in weights => writing Chinese text containing Chinese "hot". The English "hot" output is more tightly connected to and more directly derives from the physical sensation of burning. (This is of course assuming majority English training data, and in particular a relative lack of Chinese training data containing the "hot" word/concept.) So in a way, you could claim that the question of whether the word "hot" has an intrinsic meaning is relevant and useful. But it seems to me that's just one way of describing the origins of training data; use it if it's useful.