The actual paper [1] says that functional MRI (which is measuring which parts of the brain are active by sensing blood flow) indicates that different brain hardware is used for non-language and language functions. This has been suspected for years, but now there's an experimental result.
What this tells us for AI is that we need something else besides LLMs. It's not clear what that something else is. But, as the paper mentions, the low-end mammals and the corvids lack language but have some substantial problem-solving capability. That's seen down at squirrel and crow size, where the brains are tiny. So if someone figures out to do this, it will probably take less hardware than an LLM.
This is the next big piece we need for AI. No idea how to do this, but it's the right question to work on.
[1] https://www.nature.com/articles/s41586-024-07522-w.epdf?shar...
A crow has a small brain, but also has very small neurons, so ends up having 1.5B neurons, similar to a dog or some monkeys.
https://www.scientificamerican.com/article/gut-second-brain/
There are 100 million in my gut, but it doesn't solve any problems that aren't about poop, as far as I know.
https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n...
If the suspiciously round number is accurate, this puts the human gut somewhere between a golden hamster and ansell's mole-rat, and about level with a short-palated fruit bat.
Basically we need Multimodal LLM's (terrible naming as it's not an LLM then but still).
I was just pointing out that a crow's brain is built on a more advanced process node than our own. Smaller transistors.
There's been progress. Look at this 2020 work on neural net controlled drone acrobatics.[1] That's going in the right direction.
I like to think that it is my gut brain that is telling me that it's okay to have that ice cream...
Not to over-hype LLMs, but I don't see why this results says this. AI doesn't need to do things the same way as evolved intelligence has.
This (awesome!) researcher would likely disagree with what I’ve just said based on this early reference:
In the early 2000s I really was drawn to the hypothesis that maybe humans have some special machinery that is especially well suited for computing hierarchical structures.
I’d be extremely surprised if AI recapitulates the same developmental path as humans did; evolution vs. next-token prediction on an existing corpus are completely different objective functions and loss landscapes.
Similar reason we look for markers of Earth-based life on alien planets: it's the only example we've got of it existing.
Also, calling "generative grammar" productive seems wrong to me. It's been around for half a century -- what tools has it produced? At some point theory needs to come into contact with empirical reality. As far as I know, generative grammar has just never gotten to this point.
I then looked it up and they had each copy/pasted the same Stack overflow answer.
Furthermore, the answer was extremely wrong, the language I used was superficially similar to the source material, but the programming concepts were entirely different.
What this tells me is there is clearly no “reasoning” happening whatsoever with either model, despite marketing claiming as such.
Open AI O1 seems to be trained on mostly synthetic data, but it makes intuitive sense that LLMs work so well because we had the data lying around already.
The only reason humans have that "communication model" is because that's how you model other humans you speak to. It's a faculty for rehearsing what you're going to say to other people, and how they'll respond to it. If you have any profound thoughts at all, you find that your spoken language is deficient to even transcribe your thoughts, some "mental tokens" have no short phrases that even describe them.
The only real thoughts you have are non-verbal. You can see this sometimes in stupid schoolchildren who have learned all the correct words to regurgitate, but those never really clicked for them. The mildly clever teachers always assume that if they thoroughly practice the terminology, it will eventually be linked with the concepts themselves and they'll have fully learned it. What's really happening is that there's not enough mental machinery underneath for those words to ever be anything to link up with.
On the other hand, further understanding how to engage complex cognitive processes in nonverbal individuals is extremely useful and difficult to accomplish.
Proper multimodal models natively consider whatever input you give them, store the useful information in an abstracted form (i.e not just text), building it's world model, and then output in whatever format you want it to. It's no different to a mammals, just the inputs are perhaps different. Instead of relying on senses, they rely on text, video, images and sound.
In theory you could connect it to a robot and it could gather real world data much like a human, but would potentially be limited to the number of sensors/nerves it has. (on the plus side it has access to all recorded data and much faster read/write than a human).
I am a sensoral thinker, I often think and internally express myself in purely images or sounds. There are, however, some kinds of thoughts I've learned I can only fully engage with if I speak to myself out loud or at least inside of my head.
The most appropriate mode of thought depends upon the task at hand. People don't typically brag about having internal monologues. They're just sharing their own subjective internal experience, which is no less valid than a chiefly nonverbal one.
You mean besides a few layers of LLMs near input and output that deal with tokens? We have the rest of the layers.
1. Syntax
2. Semantics
3. Pragmatics
4. Semiotics
These are the layers you need to solve.
Saussure already pointed out these issues over a century ago, and Linguists turned ML Researchers like Stuart Russell and Paul Smolensky tried in vain to resolve this.
It basically took 60 years just to crack syntax at scale, and the other layers are still fairly far away.
Furthermore, Syntax is not a solved problem yet in most languages.
Try communicating with GPT-4o in colloquial Bhojpuri, Koshur, or Dogri, let alone much less represented languages and dialects.
e.g. the neural electrochemical output has a specific sequence that triggers the production of a certain hormone in your pituitary gland for e.g. and the hormone travels to the relevant body function activating/stopping it.
The most interesting thing about LLMs is probably how much relational information turns out to be encoded in large bodies of our writing, in ways that fancy statistical methods can access. LLMs aren’t thinking, or even in the same ballpark as thinking.
Warning, watch out for waving hands: The way I see it is that cognition involves forming an abstract representation of the world and then reasoning about that representation. It seems obvious that non-human animals do this without language. So it seems likely that humans do too and then language is layered on top as a turbo boost. However, it also seems plausible that you could build an abstract representation of the world through studying a vast amount of human language and that'll be a good approximation of the real-world too and furthermore it seems possible that reasoning about that abstract representation can take place in the depths of the layers of a large transformer. So it's not clear to me that we're limited by the data we have or necessarily need a different type of data to build a general AI although that'll likely help build a better world model. It's also not clear that an LLM is incapable of the type of reasoning that animals apply to their abstract world representations.
for more, see "Assembly Theory"
Feed it all the video ever recorded, hook it up to web cams, telescopes, etc. This says a lot about how the universe works, without using a single word.
When the first chess engines came out they only employed one of these: calculation. It wasn't until relatively recently that we had computer programs that could perform all of them. But it turns out that if you scale that up with enough compute you can achieve superhuman results with calculation alone.
It's not clear to me that LLMs sufficiently scaled won't achieve superhuman performance on general cognitive tasks even if there are things humans do which they can't.
The other thing I'd point out is that all language is essentially synthetic training data. Humans invented language as a way to transfer their internal thought processes to other humans. It makes sense that the process of thinking and the process of translating those thoughts into and out of language would be distinct.
Humans not taking this approach doesn’t mean that AI cannot.
If "general cognitive tasks" means "I give you a prompt in some form, and you give me an incredible response of some form " (forms may differ or be the same) then it is hard to disagree with you.
But if by "general cognitive task" you mean "all the cognitive things that human do", then it is really hard to see why you would have any confidence that LLMs have any hope of achieving superhuman performance at these things.
It could be said not as well as the ones that don't need SO.
https://arstechnica.com/ai/2024/10/llms-cant-perform-genuine...
or do you maybe think no logical reasoning is needed to do everything a human can do? Tho humans seem to be able to do logical reasoning
I think the paper should've included controls, because we don't know how strong the result is. They certainly may have proven that humans can't reason either.
Stepping back a level, it may only actually tell us that MRIs measure blood flow.
Once you've figured out how to use language, explain why this is important and to who. Then maybe what the upshot will be. The fact that someone has proven something to be true doesn't make it important.
The comment I replied to made it sound like it's important to the field of AI. It is not. Almost zero serious researchers think LLMs all by themselves are "enough". People are working on all manner of models and systems incorporating all kinds of things "not LLM". Practically no one who actually works in AI reads this paper and changes anything, because it only proves something they already believed to be true and act accordingly.
Spoiler alert: brains require a lot of blood, constantly, just to not die. Looking at blood flow on an MRI to determine neural circuitry has to deal with the double whammy of both an extremely crude tool and a correlation/causation fallacy.
This article and the study are arguably useless.
Probably by putting simulated animals into simulated environments where they have to survive and thrive.
Working at animal level is uncool, but necessary for progress. I had this argument with Rod Brooks a few decades back. He had some good artificial insects, and wanted to immediately jump to human level, with a project called Cog.[1] I asked him why he didn't go for mouse level AI next. He said "Because I don't want to go down in history as the inventor of the world's greatest artificial mouse."
Cog was a dud, and Brooks goes down in history as the inventor of the world's first good robotic vacuum cleaner.
An example was the problem of memory shared between systems. ML people started doing LLM’s with RAG. I looked into neuroscience which suggested we need a hippocampus model. I found several papers with hippocampus-like models. Combining LLM’s, vision, etc with hippocampus-like model might get better results. Rinse repeat for these other brain areas wherever we can understand them.
I also agree on testing the architectures with small, animal brains. Many do impressive behaviors that we should be able to recreate in simulators or with robotics. Some are useful, too, like how geese are good at security. Maybe embed a trained, goose brain into a camera system.
Some people will use any limitation of LLMs to deny there is anything to see here, while others will call this ‘moving the goalposts’, but the most interesting questions, I believe, involve figuring out what the differences are, putting aside the question of whether LLMs are or are not AGIs.
To some extent this is true.
To calculate A + B you could for example generate A, B for trillions of combinations and encode that within the network. And it would calculate this faster than any human could.
But that's not intelligence. And Apple's research showed that LLMs are simply inferring relationships based on the tokens it has access to. Which you can throw off by adding useless information or trying to abstract A + B.
The data itself will be most senses collecting raw data about the world most of the day for 18 years. It might require a camera on the kid’s head which I don’t like. I think people letting a team record their life is more likely. Split the project up among many families running in parallel, 1-4 per grade/year. It would probably cost a few million a year.
(Note: Parent changes might require an integration step during AI training or showing different ones in the early years.)
The training system would rapidly scan this information in. It might not be faster than human brains. If it is, we can create them quickly. That’s the passive learning part, though.
Human training involves asking lots of questions based on internal data, random exploration (esp play) with reinforcement, introspection/meditation, and so on. Self-driven, generative activities whose outputs become inputs into the brain system. This training regiment will probably need periodic breaks from passive learning to ask questions or play which requires human supervision.
Enough of this will probably produce… disobedient, unpredictable children. ;) Eventually, we’ll learn how to do AI parenting where the offspring are well-behaved, effective servants. Those will be fine-tuned for practical applications. Later, many more will come online which are trained by different streams of life experience, schooling methods, etc.
That was my theory. I still don’t like recording people’s lives to train AI’s. I just thought it was the only way to build brain-like AI’s and likely to happen (see Twitch).
My LLM concept was to do the same thing with K-12 education resources, stories, kids games, etc. Parents already could tell us exactly what to use to gradually build them up since they did that for their kids year by year. Then, several career tracts layering different college books and skill areas. I think it would be cheaper than GPT-4 with good performance.
While I generally do suspect that we need to invent some new technique in the realm of AI in order for software to do everything a human can do, I use analogies like chess engines to caution myself from certainty.
And it turns out that human brain volume and intelligence are moderately-highly correlated [1][2]!
[1]: https://pmc.ncbi.nlm.nih.gov/articles/PMC7440690/ [2]: https://www.sciencedirect.com/science/article/abs/pii/S01602...
Perhaps, but the relative success of trained LLMs acting with apparent generalised understanding may indicate that it is simply the interface that is really an LLM post training;
That the deeper into the network you go (the further from the linguistic context), the less things become about words and linguist structure specifically and the more it becomes about things and relations in general.
(This also means that multiple interfaces can be integrated, sometimes making translation possible, e.g.: image <=> tree<string>)
LLMs absolutely 100% can reason, if we take the dictionary definition; it’s trivial to show their ability to answer non-memorized questions, and the only way to do that is some sort of reasoning. I personally don’t think they’re the most efficient tool for deliberative derivation of concepts, but I also think any sort of categorical prohibition is anti-scientific. What is the brain other than a neural network?
Even if we accept the most fringe, anthropocentric theories like Penrose & Hammerhoff’s quantum tubules, that’s just a neural network with fancy weights. How could we possibly hope to forbid digital recreations of our brains from “truly” or “really” mimicking them?
The condition of “some people are bad at thing” does not equal “computer better at thing than people”, but I see this argument all the time in LLM/AI discourse.
I don't feel like this is a very meaningful argument because if you can do that generation then you must already have a superhuman machine for that task.
The whole issue with "reasoning" is that is an incompletely defined concept. Over what domain, what problem space, and what kind of experimental access do we define "reasoning"? Search is better as a concept because it comes packed with all these things, and without conceptual murkiness. Search is scientifically studied to a greater extent.
I don't think we doubt LLMs can learn given training data, we already accuse them of being mere interpolators or parrots. And we can agree to some extent the LLMs can recombine concepts correctly. So they got down the learning part.
And for the searching part, we can probably agree its a matter of access to the search space not AI. It's an environment problem, and even a social one. Search is usually more extended than the lifetime of any agent, so it has to be a cultural process, where language plays a central role.
When you break reasoning/progress/intelligence into "search and learn" it becomes much more tractable and useful. We can also make more grounded predictions on AI, considering the needs for search that are implied, not just the needs for learning.
How much search did AlphaZero need to beat us at go? How much search did humans pack in our 200K years history over 10,000 generations? What was the cost of that journey of search? That kind of questions. In my napkin estimations we solved 1:10000 of the problem by learning, search is 10000x to a million times harder.
Just a personal opinion, but in my shitty When H.A.R.L.I.E. Was One (and others) unpublished fiction pastiche (ripoff, really), I had the nascent AI stumble upon Cyc as its base for the world and "thinking about how to think."
I never thought that Cyc was enough, but I do think that something Cyc-like is necessary as a component, a seed for growth, until the AI begins to make the transition from the formally defined, vastly interrelated frames and facts in Cyc to being able to growth further and understand the much less formal knowledgebase you might find in, say Wikipedia.
Full agreement with your animal model is only sensible. If you think about macaques, they have a limited range of vocalization once they hit adulthood. Noe that the mothers almost never make a noise at their babies. Lacking language, when a mother wants to train an infant, she hurts it. (Shades of Blindsight there) She picks up the infant, grasps it firmly, and nips at it. The baby tries to get away, but the mother holds it and keeps at it. Their communication is pain. Many animals do this. But they also learn threat displays, the promise of pain, which goes beyond mere carrot and stick.
The more sophisticated multicellular animals (let us say birds, reptiles, mammals) have to learn to model the behavior of other animals in their environment: to prey on them, to avoid being prey. A pond is here. Other animals will also come to drink. I could attack them and eat them. And with the macaques, "I must scare the baby and pain it a bit because I no longer want to breastfeed it."
Somewhere along the line, modeling other animals (in-species or out-species) hits some sort of self-reflection and the recursion begins. That, I think, is a crucial loop to create the kind of intelligence we seek. Here I nod to Egan's Diaspora.
Looping back to your original point about the training data, I don't think that loop is sufficient for an AGI to do anything but think about itself, and that's where something like Cyc would serve as a framework for it to enter into the knowledge that it isn't merely cogito ergo summing in a void, but that it is part of a world with rules stable enough that it might reason, rather than "merely" statistically infer. And as part of the world (or your simulated environment), it can engage in new loops, feedback between its actions and results.
Um... What? That is a huge leap to make.
'Reasoning' is a specific type of thought process and humans regularly make complicated decisions without doing it. We uses hunches and intuition and gut feelings. We make all kinds of snap assessments that we don't have time to reason through. As such, answering novel questions doesn't necessarily show a system is capable of reasoning.
I see absolutely nothing resumbling an argument for humans having an "ineffable calculator soul", I think that might be you projecting. There is no 'categorical prohibition', only an analysis of the current flaws of specific models.
Personally, my skepticism about imminent AGI has to do believing we may be underestimating the complexity of the software running on our brain. We've reached the point where we can create digital "brains", or atleast portions of them. We may be missing some other pieces of a digital brain, or we may just not have the right software to run on it yet. I suspect it is both but that we'll have fully functional digital brains well before we figure out the software to run on them.
First, while it is a fringe idea with little backing it, it's far from the most fringe.
Secondly, it is not at all known that animal brains are accurately modeled as an ANN, any more so than any other Turing-compatible system can be modeled as an ANN. Biological neurons are themselves small computers, like all living cells in general, with not fully understood capabilities. The way biological neurons are connected is far more complex than a weight in an ANN. And I'm not talking about fantasy quantum effects in microtubules, I'm talking about well-established biology, with many kinds of synapses, some of which are "multicast" in a spatially distinct area instead of connected to specific neurons. And about the non-neuronal glands which are known to change neuron behavior and so on.
How critical any of these differences are to cognition is anyone's guess at this time. But dismissing them and reducing the brain to a bigger NN is not wise.
While I agree this is possible, I don't see why you'd think it's likely. I would instead say that I think it's unlikely.
Human communication relies on many assumptions of a shared model of the world that are rarely if ever discussed explicitly, and without which certain concepts or at least phrases become ambiguous or hard to understand.
When it comes to general intelligence, I think we are trying to run before we can walk. We can't even make a computer with a basic, animal level understanding of the world. Yet we are trying to take a tool that was developed on top of system that already had an understanding of the world and use it to work backwards to give computers an understanding of the world.
I'm pretty skeptical that we're going to succeed at this. I think you have to be able to teach a computer to climb a tree or hunt (subhuman AGI) before you can create superhuman AGI.
LLMs basically become practical when you simply scale compute up, and maybe both regions are "general compute", but language ends up on the "GPU" out of pure necessity.
So to me, these are entirely distinct questions: is the language region able to do general cognitive operations? What happens when you need to spell out "ubiquitous" or declense a foreign word in a language with declension (which you don't have memory patterns for)?
I agree it seems obvious that for better efficiency (size of training data, parameter count, compute ability), human brains use different approach than LLMs today (in a sibling comment, I bring up an example of my kids at 2yo having a better grasp of language rules than ChatGPT with 100x more training data).
But let's dive deeper in understanding what each of these regions can do before we decide to compare to or apply stuff from AI/CS.
No this is not true. For two reasons.
1. We call these things LLMs and we train it with language but we can also train it with images.
2. We also know LLMs develop a sort of understanding that goes beyond language EVEN when the medium used for training is exclusively language.
The naming of LLMs is throwing you off. You can call it a Large Language Model but this does not mean that everything about LLMs are exclusively tied only to language.
Additionally we don't even know if the LLM is even remotely similar to the way human brains process language.
No such conclusion can be drawn from this experiment.
Your responding to a claim that was never made. The claim was don't assume humans are smarter than whales. Nobody said whales are more intelligent than humans. He just said don't assume.
And yeah it seems that core primitives of intelligence exist very low in our brains. And with people like Michael Levin, there may even be a root beside nervous systems.
After all, that's what Artificial General Intelligence would at least in part be about: finding and proving new math theorems, creating new poetry, making new scientific discoveries, etc.
There is even a new challenge that's been proposed: https://arcprize.org/blog/launch
> It makes sense that the process of thinking and the process of translating those thoughts into and out of language would be distinct
Yes, indeed. And LLMs seem to be very good at _simulating_ the translation of thought into language. They don't actually do it, at least not like humans do.
Not true. You yourself have failed at reasoning here.
The problem with your logic is that you failed to identify the instances where LLMs have succeeded with reasoning. So if LLMs both fail and succeed it just means that LLMs are capable of reasoning and capable of being utterly wrong.
It's almost cliche at this point. Tons of people see the LLM fail and ignore the successes then they openly claim from a couple anecdotal examples that LLMs can't reason period.
Like how is that even logical? You have contradictory evidence therefore the LLM must be capable of BOTH failing and succeeding in reason. That's the most logical answer.
Is that the dominant chain, or is the simpler “I’ve seen animals here before that I have eaten” or “I’ve seen animals I have eaten in a place that smelled/looked/sounded/felt like this” sufficient to explain the behavior?
Higher order faculties aside, animals seem like us, just simpler.
The higher functioning ones appear to have this missing thing too. We can see it in action. Perhaps all of them do and it is just harder for us when the animal thinks very differently or maybe does not think as much, feeling more, for example.
----
Now, about that thing... and the controversy:
Given an organism, or machine for this discussion, is of sufficiently robust design and complexity that it can precisely differentiate itself from everything else, it is a being.
This thing we are missing is an emergent property, or artifact that can or maybe always does present when a state of being also presents.
We have not created a machine of this degree yet.
Mother nature has.
The reason for emergence is a being can differentiate sensory input as being from within, such as pain, or touch, and from without, such as light or motion.
Another way to express this is closed loop vs open loop.
A being is a closed loop system. It can experience cause and effect. It can be the cause. It can be the effect.
A lot comes from this closed loop.
There can be the concept of the self and it has real meaning due to the being knowing what is of itself or something, everything else.
This may be what forms consciousness. Consciousness may require a closed loop, and organism of sufficient complexity to be able to perceive itself.
That is the gist of it.
These systems we make are fantastic pieces. They can pattern match and identify relationships between the data given in amazing ways.
But they are open loop. They are not beings. They cannot determine what is part of them, what they even are,or anything really.
I am both consistently amazed and dismayed at what we can get LLM systems to do.
They are tantalizingly close!
We found a piece of how all this works and we are exploiting the cral out of it. Ok fine. Humans are really good at that.
But it will all taper off. There are real limits because we will eventually find the end goal will be to map out the whole problem space.
Who has tried computing that? It is basically all possible human thought. Not going to happen.
More is needed.
And that "more" can arrive at thoughts without having first seen a few bazillion to choose from.
Needs to be a closed loop, running on its own.
We get its attention, and it responds, or frankly if we did manage any sort of sentience, even a simulation of it, then the fact is it may not respond.
To me, that is the real test.
I am not convinced it follows. Sure LLMs don’t seem complete however there’s a lot of unspoken inference going on in LLMs that don’t map into a language directly already - the inner layers of the deep neural net that operates on abstract neurons.
Basically, if humans have had meaningful discussions about it, the product of their reasoning is there for the LLM, right?
Seems to me, the "how many R's are there in the word "strawberry" problem is very suggestive of the idea LLM systems cannot reason. If they could, the question is not difficult.
The fact is humans may never have actually discussed that topic in any meaningful way captured in the training data.
And because of that and how specific the question is, the LLM has no clear relationships to map into a response. It just does best case, whatever the math deemed best.
Seems plausible enough to support the opinion LLM'S cannot reason.
What we do know is LLMs can work with anything expressed in terms of relationships between words.
There is a ton of reasoning templates contained in that data.
Put another way:
Maybe LLM systems are poor at deduction, save for examples contained in the data. But there are a ton of examples!
So this is hard to notice.
Maybe LLM systems are fantastic at inference! And so those many examples get mapped to the prompt at hand very well.
And we do notice that and see it like real thinking, not just some horribly complex surface containing a bazillion relationships...
For that not to be the case, you'd have to take the position that humans experience consciousness and they talk about consciousness but that there is no causal link between the two! It's just a coincidence that the things you find yourself saying about consciousness line up with your internal experience?
https://www.lesswrong.com/posts/fdEWWr8St59bXLbQr/zombies-zo...
OTOH, humans (and animals) do get other data feeds (visual, context, touch/pain, smell, internal balance "sensors"...) that we develop as we grow and tie that to learning about language.
Obviously, LLMs won't replicate that since even adults struggle to describe these verbally.
Then LLMs came along, and ML folks got rather too excited that they contain implicit knowledge (which, of course, is required to deal with ambiguity). Then the new aspiration as "all in one" and "bigger is better", not analyzing what components are needed and how to orchestrate their interplay.
From an engineering (rather than science) point of view, the "end-to-end black box" approach is perhaps misguided, because the result will be a non-transparent system by definition. Individual sub-models should be connected in a way that retains control (e.g. in dialog agents, SRI's Open Agent Architecture was a random example of such "glue" to tie components together, to name but one).
Regarding the science, I do believe language adds to the power of thinking; while (other) animals can of course solve simple problems without language, language permits us to define layers of abstractions (by defining and sharing new concepts) that goes beyond simple, non-linguistic thoughts. Programming languages (created by us humans somewhat in the image of human language) and the language of mathematics are two examples where we push this even further (beyond the definition of new named concepts, to also define new "DSL" syntax) - but all of these could not come into beying without human language: all formal specs and all axioms are ultimately and can only be formulated in human language. So without language, we would likely be stuck at a very simple point of development, individually and collectively.
EDIT: 2 typos fixed
Anyway, it seems to me we are generally all in agreement (in this thread, at least), but are now being really picky about... language :)
Anyway, this is just like solipsism, you won't find a sincere one outside the asylum. Every Reddit intellectual writing such tired drivel as "who's to say humans are more intelligent than beasts?" deep down knows the score.
In my personal learning journey I have been exploring the space of intuitive learning which is dominant in physical skills. Singing requires extremely precise control of actions we can't fully articulate or even rationalise. Teaching those skills requires metaphors and visualising and a whole lot of feedback + trial & error.
I believe that this kind of learning is fundamentally non verbal and we can achieve abstraction of these skills without language. Walking is the most universal of these skills and we learn it before we can speak but if you study it (or better try to program a robot to walk with as many degrees of freedom as the human musculoskeletal system) you will discover that almost all of us don't understand what all the things that go into the "simple" task of walking!
My understanding is that people who are gifted at sports or other physical skills like musical instruments have developed the ability to discover and embed these non verbal abstractions quickly. When I practise the piano and am working on something fast, playing semiquavers at anything above 120bpm is not really conscious anymore in the sense of "press this key then that key"
The concept of arpeggio is verbal but the action is non verbal. In human thought where does verbal and non-verbal start and end? Its probably a continuum
This bias is real. Current gen ai works proportionally well the more known it is. The more training data, the better the performance. When we ask something very specific, we have the impression that it’s niche. But there is tons of training data also on many niche topics, which essentially enhances the magic trick – it looks like sophisticated reasoning. Whenever you truly go “off the beaten path”, you get responses that are (a) nonsensical (illogical) and (b) “pulls” you back towards a “mainstream center point” so to say. Anecdotally of course..
I’ve noticed this with software architecture discussions. I would have some pretty standard thing (like session-based auth) but I have some specific and unusual requirement (like hybrid device- and user identity) and it happily spits out good sounding but nonsensical ideas. Combining and interpolating entirely in the the linguistic domain is clearly powerful, but ultimately not enough.
Solving puzzles is a specific cognitive task, not a general one.
Language is a continuum, not a puzzle. The problem with LLMs is that testing has been reduced to performance on language puzzles, mostly with hard edges - like bar exams, or letter counting - and they're a small subset of general language use.
Imagine trying to limit, control, or explain a being without familiar cognitive structures.
Is there a reason to care about such unfamiliar modalities of cognition?
Based on my experience with toddlers, a rather smart dog, and my own thought processes, I disagree that language is a fundamental component of abstraction. Of sharing abstractions, sure, but not developing them.
When I'm designing a software system I will have a mental conception of the system as layered abstractions before I have a name for any component. I invent names for these components in order to define them in the code or communicate them to other engineers, but the intuition for the abstraction comes first. This is why "naming things" is one of the hard problems in computer science—because the name comes second as a usually-inadequate attempt to capture the abstraction in language.
As many people have pointed out, Searle's argument begs the question by tacitly assuming that if anything about the room understands Chinese, it can only be the person within it.
Because whales or dolphins didn’t evolve hands. Hands are a foundational prerequisite for building technology. So if whales or dolphins had hands we don’t know if they would develop technology that can rival us.
Because we don’t know, that’s why he says don’t assume. This isn’t a “deep down we know” thing like your more irrational form of reasoning. It is a logical conclusion: we don’t know. So don’t assume.
Music or sports are more interesting to investigate (in my opinion) since those specific actions won’t be preprogrammed and must be learned independently.
The same way we build abstractions for language in order to perform “telepathy” it seems like for music or sports we build body-specific abstractions. They work similar to words within our own brain but are not something easily communicated since they’re not tied to any language, it’s just a feeling.
I think it’s an interesting point that quite often the best athletes or musicians are terrible coaches. They probably have a much more innate internal language for their body that cannot be communicated easily. Partially, I think, that their body is more different than others which helps them be exceptional. Or that weaker athletes or musicians need to focus much more on lessons from others, so their body language gets tied much closer to human language and that makes it much easier for them to then communicate the lessons they learn to others.
To deal with the awkwardly apparent fact that consciousness certainly seems to have physical effects, zombiephiles challenge the notion that physics is causally closed, so that it is conceivable that something non-physical can cause physical effects. Their approach is to say that the causal closure of physics is not provable, but at this point, the argument has become a lexicographical one, about the definition of the words 'physics' and 'physical' (if one insists that 'physical' does not refer to a causally-closed concept, then we still need a word for the causal closure within which the physical is embedded - but that's just what a lot of people take 'physical' to mean in the first place.) None of the anti-physicalists have been able, so far, to shed any light on how the mind is causally effective in the physical world.
You might be interested in the late Daniel Dennett's "The Unimagined Preposterousness of Zombies": https://dl.tufts.edu/concern/pdfs/6m312182x
The "they MIGHT be as intelligent, just lacking hands" theory can't have the same weight as "nah" in an honest mind seeing the overwhelming clues (yes, not proof, if that's what you want) against it. Again, same way that you can't disprove solipsism.
'Reasoning' is a specific type of thought process
I personally have lots of thoughts on the topic and look to Kant and Hegel for their definitions of reason as the final faculty of human cognition (after sensibility, understanding, and judgement), and I even think there’s good reason (heh) to think that LLMs are not a great tool for that on their own. But my point is that none of the LLM critics have a definition anywhere close to that level of specificity.
Usually, “reason” is used to mean “good cognition”, so “LLMs can’t reason” is just a variety of cope/setting up new goalposts. We all know LLMs aren’t flawless or infinite in their capabilities, but I just don’t find this kind of critique specific enough to have any sort of scientific validity. IMHO
Generally, I absolutely agree that he is not humble in the sense of expressing doubt about his strongly held beliefs. He’s been saying pretty much the same things for decades, and does not give much room for disagreement (and ofc this is all ratcheted up in intensity in his political stances). I’m using humble in a slightly different way, tho: he insists on qualifying basically all of his statements about archaeological anthropology with “we don’t have proof yet” and “this seems likely”, because of his fundamental belief that we’re in a “pre-Galilean” (read: shitty) era of cognitive science.
In other words: he’s absolutely arrogant about his core structural findings and the utility of his program, but he’s humble about the final application of those findings to humanity.
ultimately, there's no reason that a general algorithm couldn't do the job of a specific one, just less efficiently.
All intelligence is the mitigation of uncertainty (the potential distributed problem.) if it does not mitigate uncertainty it is not intelligence, it is something else.
Intelligence is a technology. For all life intelligence and the infrastructure of performing work efficiently (that whole entropy thing again) is a technology. Life is an arms race to maintain continuity (identity, and the very capacity of existential being.)
The modern problem is achieving reliable behavioral intelligence (constrained to a specific problem domain.) AGI is a phantasm. What manifestation of intelligence appears whole and complete and is always right? These are the sorts of lies you tell yourself, the ones that get you into trouble. They distract from tangible real world problems, perhaps causing some of them. True intelligence is a well calibrated “scalar” domain specific problem (uncertainty) reducer. There are few pressing idempotent obstructions in the real world.
Intelligence is the mitigation of uncertainty.
Uncertainty is the domain of negative potential (what,where,why,how?)
Mitigation is the determinant resolve of any constructive or destructive interference affecting (terminal resolve within) the problem domain.
Examples of this may be piled together mountains high, and you may call that functional AGI, though you would be self deceiving. At some point “good enough” may be declared for anything so passing as yourselves.
As humanity has struggled to understand the world, it has frequently given names to concepts that seem to matter, well before it is capable of explaining with any sort of precision what these things are, and what makes them matter - take the word 'energy', for example.
It seems clear to me that one must have these vague concepts before one can begin to to understand them, and also that it would be bizarre not to give them a name at that point - and so, at that point, we have a word without a locked-down definition. To insist that we should have the definition locked down before we begin to investigate the phenomenon or concept is precisely the wrong way to go about understanding it: we refine and rewrite the definitions as a consequence of what our investigations have discovered. Again, 'energy' provides a useful case study for how this happens.
A third point about the word 'energy' is that it has become well-defined within physics, and yet retains much of its original vagueness in everyday usage, where, in addition, it is often used metaphorically. This is not a problem, except when someone makes the lexicographical fallacy of thinking that one can freely substitute the physics definition into everyday speech (or vice-versa) without changing the meaning.
With many concepts about the mental, including 'reasoning', we are still in the learning-and-writing-the-definition stage. For example, let's take the definition you bring up: reasoning as good cognition. This just moves us on to the questions of what 'cognition' means, and what distinguishes good cognition from bad cognition (for example, is a valid logical argument predicated on what turns out to be a false assumption an example of reasoning-as-good-cognition?) We are not going to settle the matter by leafing through a dictionary, any more than Pedro Carolino could write a phrase book just from a Portugese-English dictionary (and you are probably aware that looking up definitions-of-definitions recursively in a dictionary often ends up in a loop.)
A lot of people want to jump the gun on this, and say definitively either that LLMs have achieved reasoning (or general intelligence or a theory of mind or even consciousness, for that matter) or that they have not (or cannot.) What we should be doing, IMHO, is to put aside these questions until we have learned enough to say more precisely what these terms denote, by studying humans, other animals, and what I consider to be the surprising effectiveness of LLMs - and that is what the interviewee in the article we are nominally discussing here is doing.
You entered this thread by saying (about the paper underlying an article in Ars Tech [1]) I’ll pop in with a friendly “that research is definitely wrong”. If they want to prove that LLMs can’t reason..., but I do not think there is anything like that claim in the paper itself (one should not simply trust what some person on HN says about a paper. That, of course, goes as much for what I say about it as what the original poster said.) To me, this looks like the sort of careful, specific and objective work that will lead to us a better understanding of our concepts of the mental.
The absence of both of these things is an incredible crippler for technological development. It doesn't matter how intelligent you are, you're never going to achieve much technologically without these.
I don't think brain size correlations is as straightforward as 'bigger = better' every time but we simply don't know how intelligent most of these species are. Land and Water are completely different beasts.
You would think the whole "split-brain" thing would have been the first clue; apparently not.
We have, it's called DreamCoder. There's a paper and everything.
Everything needed for AGI exists today, people simply have (incorrect) legacy beliefs about cognition that are holding them back (e.g. "humans are rational").
We need to add the 5 senses, of which we have now image, audio and video understanding in LLMs. And for agentic behavior they need environments and social exposure.
Intelligence is the ability to use experience to predict your environment and the outcomes of your own actions. It's a tool for survival.
A black box that works in human language and can be investigated with perturbations, embedding visualizations and probes. It explains itself as much ore more than we can.
However, I do think that a meaningful intelligence comparison between humans and dolphins, etc, would conclude that we are more intelligent, especially based on our reasoning/planning (= multi-step prediction) abilities, which allows us not only to predict our environment but also to modify it to our desires in very complex ways.
Remember in CS theory, a language is just a set of strings. If you think in pictures that is STILL a language if your pictures are structured.
So I'm really handwaving the above just to suggest that it all depends on the assumptions that each expert is making in elucidating this debate which has a long history.
I'm not sure how you would make meaningful comparisons here. We can't communicate to them as they communicate and we live in almost completely different environments. Any such comparison would be extremely biased to us.
>which allows us not only to predict our environment but also to modify it to our desires in very complex ways.
We modify our environment mostly through technology. Intelligence is a big part of technology sure but it's not the only part of it and without the other parts (hands with opposable thumbs, fire etc), technology as we know it wouldn't exist and our ability to modify the environment would seem crippled to any outside observer regardless of how intelligent we may be.
It's not enough to think that the earth revolves around the sun, we need to build the telescopes (with hands and materials melted down and forged with fire) to confirm it.
It's not enough to dream and devise of flight, we need the fire to create the materials that we dug with our hands and the hands to build them.
It's not enough to think that Oral communication is insufficient for transmitting information through generations. What else will you do without opposable thumbs or an equivalent ?
Fire is so important for so many reasons but one of the biggest is that it was an easy source of large amounts of energy that allowed us to bootstrap technology. Where's that easy source of energy underwater ?
Without all the other aspects necessary for technology, we are relegated to hunter/gatherer levels of influencing the environment at best. Even then, we still crafted tools that creatures without opposable thumbs would never be able to craft.
Despite being an LLM skeptic of sorts, I don’t think that necessarily follows. The LLM matrix multiplication machinery may well be implementing an equivalent of the human non-language cognitive processing as a side effect of the training. Meaning, what is separated in the human brain may be mixed together in an LLM.
Unfortunately, you won't get one. We simply don't know enough about cognition to create rigourous definitions of the type you are looking for.
Instead, this paper, and the community in general are trying to perform practical capability assessments. The claim that the GSM8k measures "mathematical reasoning" or "logical reasoning" didn't come from the skeptics.
Alan Turring didn't try to define intelligence, he created a practical test that he thought would be a good benchmark. These days we believe we have better ones.
> I just don’t find this kind of critique specific enough to have any sort of scientific validity. IMHO
"Good cognition" seems like dismisal of a definition, but this is exactly the definition that the people working on this care about. They are not philosphers, they are engineers who are trying to make a system "better" so "good cognition" is exactly what they want.
The paper digs into finding out more about what types of changes impacts peformance on established metrics. The "noop" result is pretty interesting since "relevancy detection" isn't something we commonly think of as key to "good cognition", but a consequence of it.
Unless we're getting metaphysical to the point of describing quantum systems as possessig a language, there are various continuous analog systems that can compute without a formal grammar. The language system could be the one that thinks in discrete 'tokens'; the conscious system something more complex.
At least to our own perception, and degree of understanding, it would appear that the ocean habitat(s) of dolphins are far less diverse and demanding. Evidentially complex enough to drive their intelligence though, so perhaps we just don't understand the complexity of what they've evolved to do.
Apple’s recent research summarized here [0] is worth a read. In short, they argue that what LLMs are doing is more akin to advanced pattern recognition than reasoning in the way we typically understand reasoning.
By way of analogy, memorizing mathematical facts and then correctly recalling these facts does not imply that the person actually understands how to arrive at the answer. This is why “show your work” is a critical aspect of proving competence in an education environment.
An LLM providing useful/correct results only proves that it’s good at surfacing relevant information based on a given prompt. That fact that it’s trivial to cause bad results by making minor but irrelevant changes to a prompt points to something other than a truly reasoned response, i.e. a reasoning machine would not get tripped up so easily.
An easy conclusion to jump to but I believe we need to be more careful. Nothing in these findings proves conclusively that non-verbal reasoning mechanism equivalent to humans couldn't evolve in some part of a sufficiently large ANN trained on text and math. Even though verbal and non-verbal reasoning occurs in two distinct parts of the brain, it doesn't mean they're not related.
It’s bloody obvious that when I classify success I mean that the llm is delivering a correct and unique answer for a novel prompt that doesn’t exist in the original training set. No need to go over the same tired analogies that have been regurgitated over and over again that you believe LLMs are reusing memorized answers. It’s a stale point of view. The overall argument has progressed further then that and we now need more complicated analysis of what’s going on with LLMs
Sources: https://typeset.io/papers/llmsense-harnessing-llms-for-high-...
https://typeset.io/papers/call-me-when-necessary-llms-can-ef...
And these two are just from a random google search.
I can find dozens and dozens of papers illustrating failures and successes of LLMs which further nails my original point. LLMs both succeed and fail at reasoning.
The main problem right now is that we don’t really understand how LLMs work internally. Everyone who claims they know LLMs can’t reason are just making huge leaps of irrational conclusions because not only does their conclusion contradict actual evidence but they don’t even know how LLMs work because nobody knows.
We only know how LLMs work at a high level and we only understand these things via the analogy of a best fit curve in a series of data points. Below this abstraction we don’t understand what’s going on.
Anything that doesn't have a spine, I'm pretty sure.
Also if we look at just auditory, tons of creatures are deaf and don't need that.
> Imagine trying to limit, control, or explain a being without familiar cognitive structures.
I don't see why any of that that affects whether it's intelligent.
I'm not sure why everyone assumes an AGI would just automatically do creativity considering most people are not very creative, despite them quite literally being capable, most people can't create anything. Why wouldn't an AGI have the same issues with being "awake" that we do? Being capable of knowing stuff - as you pointed out, far more facts than a person ever could, I think an awake AGI may even have more "issues" with the human condition than us.
Also - say an AGI comes into existence that is awake, happy and capable of truly original creativity - why tf does it write us poetry? Why solve world hunger - it doesn't hunger. Why cure cancer - what can cancer do to to it?
AGI as currently envisioned is a mythos of fantasy and science fiction.
An Ab Initio AGI would maybe be free of our legacy, but LLMs certainly are not.
I would expect a ship-like intelligence a la the Culture novels to have non-English based cognition. As far as we can tell, our own language generation is post-hoc explanation for thought more so than the embodiment of thought.
The first three paragraphs you wrote very succinctly and obviously summarize the fundamental flaw of our modern science - that it can't make leaps, at all.
There is no leap of faith in science but there is science that requires such leaps.
We are stuck bc those most capable of comprehending concepts they don't understand and are unexplainable - they won't allow themselves to even develop a vague understanding of such concepts. The scientific method is their trusty hammer and their faith in it renders all that isn't a nail unscientific.
Admitting that they don't kno enough would be akin to societal suicide of their current position - the deciders of what is or isn't true, so I don't expect them to withhold their conclusions til they are more able to.
They are the "priest class" now ;)
I agree with your humble opinion - there is much more we could learn if that was our intent and considering the potential of this, I think we absolutely ought to make certain that we do everything in our power to attain the best possible outcomes of these current and future developments.
Transparent and honest collaboration for the betterment of humanity is the only right path to an AGI god - to oversimplify a lil bit.
Very astute, well formulated position, presented in accessible language and with humility even!
Well done.
That said, i definitely would not say the Ocean is particularly less diverse or demanding.
Even with our limited understanding, there must be adaptations for Pressure, Salinity, light, Energy, Buoyancy, Underwater Current etc that all vary significantly by depth and location.
And the bottlenose dolphin for instance lives in every ocean of the world except the Arctic and the Antarctic oceans.
(also important to note that NNs/LLMs operate on... abstract vectors, not "language" -- not relevant as a response to your post though).
The evidence is using one instance of the LLM parroting training data while completely ignoring contradicting evidence where the LLM created novel answers to novel prompts out of thin air.
>Observations trump claims.
No. The same irrational hallucinations that plague LLMs are plaguing human reasoning and trumping rational thinking.
Right, but big brains do actively impede you - they require a lot of energy, so there needs to be some offsetting benefit.
Other examples exist.
[0]That example is due to tokenization. DoH! I knew better too.
Ah well.
Presumably they have some sort biological input processing or sensory inputs. They don't eat data.
Right, and this is why claims that models are “reasoning” can’t be taken at face value. This space is filled with overloaded terms and anthropomorphic language that describes some behavior of the LLM but this doesn’t justify a leap to the belief that these terms actually represent the underlying functionality of the model, e.g. when terms like “hallucinate”, “understand”, etc. are used, they do not represent the biological processes these ideas stem from or carry the implications of a system that mimics those processes.
> Everyone who claims they know LLMs can’t reason are just making huge leaps of irrational conclusions because not only does their conclusion contradict actual evidence but they don’t even know how LLMs work because nobody knows.
If you believe this to be true, you must then also accept that it’s equally irrational to claim these models are actually “reasoning”. The point of citing the Apple paper was that there’s currently a lack of consensus and in some cases major disagreement about what is actually occurring behind the scenes.
Everything you’ve written to justify the idea that reasoning is occurring can be used against the idea that reasoning is occurring. This will continue to be true until we gain a better understanding of how these models work.
The reason the Apple paper is interesting is because it’s some of the latest writing on this subject, and points at inconvenient truths about the operation of these models that at the very least would indicate that if reasoning is occurring, it’s extremely inconsistent and unreliable.
No need to be combative here - aside from being against HN guidelines, there just isn’t enough understanding yet for anyone to be making absolute claims, and the point of my comment was to add counterpoints to a conversation, not make some claim about the absolute nature of things.
If a novel low probability conclusion that is correct was arrived at from a novel prompt where neither the prompt nor the conclusion existed in the training set, THEN by logic the ONLY possible way the conclusion was derived was through reasoning. We know this, but we don't know HOW the model is reasoning.
The only other possible way that an LLM can arrive at low probability conclusions is via random chance.
>The point of citing the Apple paper was that there’s currently a lack of consensus and in some cases major disagreement about what is actually occurring behind the scenes.
This isn't true. I quote the parent comment:
"What this tells me is there is clearly no “reasoning” happening whatsoever with either model, despite marketing claiming as such."
>Everything you’ve written to justify the idea that reasoning is occurring can be used against the idea that reasoning is occurring. This will continue to be true until we gain a better understanding of how these models work.
Right and I took BOTH pieces of contradictory evidence into account and I ended up with the most logical conclusion. I quote myself:
"You have contradictory evidence therefore the LLM must be capable of BOTH failing and succeeding in reason. That's the most logical answer."
Right. And this, again, was my conclusion. But I took it a bit further. Read again what I said in the first paragraph of this very response.
>No need to be combative here - aside from being against HN guidelines, there just isn’t enough understanding yet for anyone to be making absolute claims, and the point of my comment was to add counterpoints to a conversation, not make some claim about the absolute nature of things.
You're not combative and neither am I. I respect your analysis here even though you dismissed a lot of what I said (see quotations) and even though I completely disagree and I believe you are wrong.
I think there's a further logical argument you're not realizing and I pointed it out in the first paragraph. LLMs are arriving at novel answers from novel prompts that don't exist in the data set. These novel answers have such low probability of existing via random chance that the ONLY other explanation for it is covered by the broadly defined word: "reasoning".
Again, there is also evidence of prompts that aren't arrived at via reasoning, but that doesn't negate the existence of answers to prompts that can only be arrived via reasoning.
Contrast to the statistical approach. It's easy to point to something like Google translate. If Chomsky's approach gave us a tool like that, I'd have no complaint. But my sense is that it just hasn't panned out.
> Recent work has revealed that the neural activity patterns correlated with sensation, cognition, and action often are not stable and instead undergo large scale changes over days and weeks—a phenomenon called representational drift.
[...]
So, I'm not sure how conclusive this fmri activation study is either.
Though, is there a proto language that's not even necessary for the given measured aspects of condition?
Which artificial network architecture best approximates which functionally specialized biological neutral networks?
OpenCogPrime:KnowledgeRepresentation > Four Types of Knowledge: https://wiki.opencog.org/w/OpenCogPrime:KnowledgeRepresentat... :
> Sensory, Procedural, Episodic, Declarative
From https://news.ycombinator.com/item?id=40105068#40107537 re: cognitive hierarchy and specialization :
> But FWIU none of these models of cognitive hierarchy or instruction are informed by newer developments in topological study of neural connectivity;
There is no reason to assume consciousness is Turing computable [1].
[1] https://en.m.wikipedia.org/wiki/Church%E2%80%93Turing_thesis
I suppose one could build an LLM around a lora that's being continuously trained to attempt to get it to adapt to new scenarios.
I think it's something like the counting parts of problems that current models are shaky with, and I imagine it's a training data problem.
[1] Mark Bickford, Liron Cohen, Robert L. Constable, and Vincent Rahli. 2018. Computability Beyond Church-Turing via Choice Sequences. In Proceedings of the 33rd Annual ACM/IEEE Symposium on Logic in Computer Science (LICS '18). Association for Computing Machinery, New York, NY, USA, 245–254. https://doi.org/10.1145/3209108.3209200