Basically we need Multimodal LLM's (terrible naming as it's not an LLM then but still).
There's been progress. Look at this 2020 work on neural net controlled drone acrobatics.[1] That's going in the right direction.
This (awesome!) researcher would likely disagree with what I’ve just said based on this early reference:
In the early 2000s I really was drawn to the hypothesis that maybe humans have some special machinery that is especially well suited for computing hierarchical structures.
Also, calling "generative grammar" productive seems wrong to me. It's been around for half a century -- what tools has it produced? At some point theory needs to come into contact with empirical reality. As far as I know, generative grammar has just never gotten to this point.
The only reason humans have that "communication model" is because that's how you model other humans you speak to. It's a faculty for rehearsing what you're going to say to other people, and how they'll respond to it. If you have any profound thoughts at all, you find that your spoken language is deficient to even transcribe your thoughts, some "mental tokens" have no short phrases that even describe them.
The only real thoughts you have are non-verbal. You can see this sometimes in stupid schoolchildren who have learned all the correct words to regurgitate, but those never really clicked for them. The mildly clever teachers always assume that if they thoroughly practice the terminology, it will eventually be linked with the concepts themselves and they'll have fully learned it. What's really happening is that there's not enough mental machinery underneath for those words to ever be anything to link up with.
Proper multimodal models natively consider whatever input you give them, store the useful information in an abstracted form (i.e not just text), building it's world model, and then output in whatever format you want it to. It's no different to a mammals, just the inputs are perhaps different. Instead of relying on senses, they rely on text, video, images and sound.
In theory you could connect it to a robot and it could gather real world data much like a human, but would potentially be limited to the number of sensors/nerves it has. (on the plus side it has access to all recorded data and much faster read/write than a human).
I am a sensoral thinker, I often think and internally express myself in purely images or sounds. There are, however, some kinds of thoughts I've learned I can only fully engage with if I speak to myself out loud or at least inside of my head.
The most appropriate mode of thought depends upon the task at hand. People don't typically brag about having internal monologues. They're just sharing their own subjective internal experience, which is no less valid than a chiefly nonverbal one.
e.g. the neural electrochemical output has a specific sequence that triggers the production of a certain hormone in your pituitary gland for e.g. and the hormone travels to the relevant body function activating/stopping it.
Generally, I absolutely agree that he is not humble in the sense of expressing doubt about his strongly held beliefs. He’s been saying pretty much the same things for decades, and does not give much room for disagreement (and ofc this is all ratcheted up in intensity in his political stances). I’m using humble in a slightly different way, tho: he insists on qualifying basically all of his statements about archaeological anthropology with “we don’t have proof yet” and “this seems likely”, because of his fundamental belief that we’re in a “pre-Galilean” (read: shitty) era of cognitive science.
In other words: he’s absolutely arrogant about his core structural findings and the utility of his program, but he’s humble about the final application of those findings to humanity.
Contrast to the statistical approach. It's easy to point to something like Google translate. If Chomsky's approach gave us a tool like that, I'd have no complaint. But my sense is that it just hasn't panned out.