To AI skeptics bristling at these numbers, I’ve got a potentially controversial question: what’s the difference between this and the scientific consensus on Climate Change? Why heed the latter and not the former?
Maybe I'm overly optimistic (or pessimistic depending on your point of view, I suppose), but 50% by 2047 seems low to me. That just feels like an eternity of development, and even if we maintain the current pace (let alone see it accelerate as AI contributes more to its own development), it's difficult for me to imagine what humans will still be better able to do than AI in over a decade.
I do wonder if the question is ambiguously phrased and some people interpreted it as pure AI (i.e. just bits) while others answered it with the assumption that you'd also have to have the sort of bipedal robot enabled with AI that would allow it to take on all the manual tasks humans do.
You might as well roll a ball down an incline and then ask me whether Keynes was right.
Some interesting work here on using LLMs to improve on open-domain robotics: https://arstechnica.com/information-technology/2023/03/embod...
If you mean the last year, is that pace maintainable?
I say this as someone who written several pieces about xrisk from AI, and who is concerned. The models and reasoning are simply not nearly as detailed or well-tested as in the case of climate.
What’s the progression that leads to AI human extinction ?
* It also goes without saying that by this definition I mean to say that humanity will no longer be able to meaningfully help in any qualitative way with respect to intellectual tasks (e.g. AGI > human; AGI > human + computer; AGI > human + internet; AGI > human + LLM).
Fundamentally I believe AGI will never happen without a body. I believe intelligence requires constraints and the ultimate constraint is life. Some omniscient immortal thing seems neat, but I doubt it'll be as smart since it lacks any constraints to drive it to growth.
It needs vast resources to operate. As the competition in AI heats up, it will continually have to create new levels of value to survive.
Not making any predictions about OpenAI, except that as its machines get smarter, they will also get more explicitly focused on its survival.
(As apposed to the implicit contribution of AI to its creation of value today. The AI is in a passive role for the time being.)
I don't think their results are meaningful at all.
Asking random AI researchers about automating a field they have no idea about means nothing. What do I know about the job of a surgeon? My opinion on how current models can automate a job I don't understand is worthless.
Asking random AI researchers about automation outside of their area of expertise is also worthless. A computer vision expert has no idea what the state of the art in grasping is. So what does their opinion on installing wiring in a house count for? Nothing.
Even abstract tasks like translation. If you aren't an NLP researcher who has dealt with translation you have no idea how you even measure how good a translated document is, so why are you being asked when translation will be "fluent"? You're asking a clueless person a question they literally cannot even understand.
This is a survey of AI hype, not any indication of what the future holds.
Their results are also highly biased. Most senior researchers aren't going to waste their time filling this out (90% of people did not fill it out). They almost certainly got very junior people and those with an axe to grind. Many of the respondents also have a conflict of interest, they run AI startups. Of course they want as much hype as possible.
This is not a survey of what the average AI researcher thinks.
That bar is insane. By that logic, humans aren't intelligent.
I believe AGI must be definitionally superior. Anything else and you could argue it’s existed for a while, e.g. computers have been superior at adding numbers basically their entire existence. Even with reasoning, computers have been better for a while. Language models have allowed for that reasoning to be specified in English, but you could’ve easily written a formally verified program in the 90s that exhibits better reasoning in the form of correctness for discrete tasks.
Even with game playing Go, and Chess, games that require moderate to high planning skills are all but solves with computers, but I don’t consider them AGI.
I would not consider N entities that can each beat humanity in the Y tasks humans are capable of to be AGI, unless some system X is capable of picking N for Y as necessary without explicit prompting. It would need to be a single system. That being said I could see one disagreeing haha.
I am curious if anyone has different definition of AGI that cannot already be met now.
The approach was tremendously simple and totally naive, but it was still interesting. At the time a supercomputer could simulate the full brain of a flatworm. We then simply applied a Moore's law-esque approach of assuming simulation capacity can double every 1.5-2 years (I forget the time period we used), and mapped out different animals that we had the capability to simulate on each date. We showed years for a field mouse, a corvid, a chimp, and eventually a human brain. The date we landed on was 2047.
There are so many things wrong with that approach I can't even count, but I'd be kinda smitten if it ended up being correct.
Or a group of millions of such AGI instances in a similar time frame?
This paper might be helpful for understanding the nervous system in particular:
https://royalsocietypublishing.org/doi/10.1098/rstb.2017.037...
Or maybe I'm just jaded after a couple decades of consistently underbidding engineering and software projects :)
edit: Fix typo
If they accelerate the burning of fossil fuels, extract and process minerals on land and in the ocean without concern for pollution, replace large areas of the natural world with solar panels, etc., the world could rapidly become hostile for large creatures.
An ocean die out as a result of massive deep sea mining would be particularly devastating. It's very hard to contain pollution in the ocean.
Same for lakes. And without clean water things will get bad everywhere.
Ramping up the frequency of space launches a few orders of magnitude into the solar system for further resources could heavily pollute the atmosphere.
Microbes might be fine, and be able to evolve to changes, for much longer.
1. Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works.
2. Anything that's invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.
3. Anything invented after you're thirty-five is against the natural order of things.”
― Douglas Adams, The Salmon of Doubt
We can't even simulate all of the chemical processes inside a single cell. We don't even know all of the chemical processes. We don't know the function of most proteins.
I'm inclined to believe this as well, but rather than "it won't happen", I take it to mean that AI and robotics just need to unify. That's already starting to happen.
You're basically requiring AGI to be smarter/better than the smartest/best humans in every single field.
What you're describing is ASI.
If we have AGI that is on the level of an average human (which is pretty dumb), it's already very useful. That gives you robotic paradise where robots do ALL mundane tasks.
log(10^17/10^12)/log(2) = 16.61 so assuming 1.5 years per doubling, that'll be another 24.9 years - December, 2048 - before 8x X100s can simulate the human brain.
And "brain in a jar" is different from "AGI"
Brain > Cell > Molecules(DNA and otherwise) > Atoms > Sub-atomic particles...
Potentially dumb question, but how deeply do we need to understand the underlying components to simulate a flatworm brain?
I think this is very plausible--that AI won't really be AGI until it has a way to physically grow free from the umbilical chord that is the chip fab supply chain.
So it might take Brainoids/Brain-on-chip technology to get a lot more advanced before that happens. However, if there are some breakthroughs in that tech, so that a digital AI could interact with in vitro tissue, utilize it, and grow it, it seems like the takeoff could be really fast.
There may be (almost certainly is) a more optimized way a general intelligence could be implemented, but we can't confidentally say what that requires.
You might be right, but this is the kind of hubris that is often embarrassing in hindsight. Like when Aristotle thought the brain was a radiator.
That's because we define "general intelligence" circularly as "something the human brain does."
I suspect if you just want an automaton that provides the utility of a human brain, we'll be fine just using statistical approximations based on what we see biological neurons doing. The utility of LLMs so far has moved the needle in that direction for sure, although there's still enough we don't know about cognition that we could still hit a surprise brick wall when we start trying to build GPT-6 or whatever. But even so, a prediction of 2047 for that kind of AGI is plausible (ironically, any semblance of Moore's Law probably won't last until then).
On the other hand, if you want to model a particular human brain... well, then things get extremely hairy scientifically, philosophically, and ethically.
Citation needed?
As long as it's modern scientific evidence and not a 2,300 year old anecdote, of course.
So it is not unreasonable to expect I can have an Ana de Armas AI in 2049?
I hope you AI people are better than the flying car people.
We have almost no idea what biological neurons are doing, or why. At least we didn't when I got my PhD in neuroscience a little over 10 years ago. Maybe it's a solved problem by now.
That would still require a lot of equipment potentially but it's there.
Another possibility would be some kind of rogue agent scenario where the program and hide and distribute itself on many machines, and interact with people to get them to do bad things or give it money. I think someone already demonstrated one of the LLMs doing some kind of social engineering attack somewhere and getting the support agent to let them in. Not hard to imagine some kind of government funded weapon that scales up that kind of attack. Imagine whole social movements, terrorist groups, or religious cults run by an autonomous agent.
In this light, widespread acknowledgement of xrisk will only come once we have a statistical model that shows it will. And at that point, it seems like it would be too late… Perhaps “Intelligence Explosion Modeling” should be a new sub-field under “AI Safety & Alignment” — a grim but useful line of work.
FAKE_EDIT: In fact, after looking it up, it sorta is! After a few minutes skimming I recommend Intelligence Explosion Microeconomics (Yudkowsky 2013>) to anyone interested in the above. On the pile of to-read lit it goes…
I think current AI research has shown that simply representing a brain as a neural network (e.g. fully connected, simple neurons) is not sufficient for AGI.
It's tempting to look at Moore's law and use say the development of the 8080, z-80 and 6502 in 1975 as an epoch. But it's hard to use that to get a visceral sense of how much things changed. I think RAM - in other words, available memory - may be more helpful, and it does relate in a distant way with model size and available GPU memory.
So the question is, if we surveyed a group of devs, engineers and computer scientists in 1975 and asked them to extrapolate and predict available RAM a few decades out, how well would their predictions map to reality?
In 1975 the Altair 8800 microcomputer with the 8080 processor had 8K of memory for the high end kit (4096 words).
8 years later, in 1983 the Apple IIe (which I learned to program on) had 64K RAM as standard, or 8 times the RAM.
13 years later in 1996, 16 to 32 MB was fairly commonplace in desktop PCs. That's 32,768K which is 4096 times the 8K available 21 years earlier.
30 years later in 2005, it wasn't unusual to find 1GB of RAM or 1,048,576K or 131,072 times 8K from 30 years earlier.
Is it realistic to expect a 1975 programmer, hardware engineer or computer scientist to predict that available memory in a desktop machine will be over 100,000 times greater 30 years in the future? We're not even taking into account moving from byte oriented CPUs to 32bit CPUs, or memory bandwidth.
2054 is 30 years in the future. It's going to fly by. I think given the unbelievable rate of change we've seen in the past, and how it accelerates, any prediction today from the smartest and most forward thinking people in AI will vastly underestimate what 2054 will look like.
Edit: 32bit CPU's not 64. Typo.
It seems clear at this point that although computers can be made to model physical systems to great degree, this is not the area where they naturally excel. Think of modeling the temperature of a room, you could try and recreate the physically accurate simulation of every particle and its velocity. We could then create better software to model the particles on ever more powerful and specific hardware to model bigger and bigger rooms.
Just like how thermodynamics might make more sense to model statistically, I think intelligence is not best modeled at the synapse layer.
I think the much more interesting question is what would the equivalent of a worm brain be for a digital intelligence?
If I remember an article from a few days ago correctly, this would make the AI threat an “uncertain” one, rather than merely “risky” like climate change (we know what might happen, we just need to figure out how likely it is).
EDIT: Disregarding the fact that in that article, climate change was actually the example of a quintessentially uncertain problem… makes me chuckle. A lesson on relative uncertainty
- being a roughly human equivalent remote worker.
- having robust common sense on language tasks
- having robust common sense on video, audio and robotics tasks, basically housework androids (robotics is not the difficulty anymore).
Just to name a few. There is a huge gap between what LLMs van do and what you describe!
I assure you computers already are superior to a human remote worker whose job it is to reliably categorize items or to add numbers. Look no further than the duolingo post that's ironically on the front page at the time of this writing with this very post.
computers have been on par with human translators at some languages since the 2010s. an hypothetical AGI is not god, it still would need exposure, similar to training with LLMs. We're already near the peak with respect to that problem.
I'm not familiar with a "hard turing test." What is that?
as I mention in another post, this is why I do not make any distinction between AGI and superintelligence. I believe they are the same thing. a thought experiment - what would it mean for a human to be superintelligent? presumably it would mean learning things with the least possible amount of exposure (not omniscience, necessarily).
In many ways AI risk looks like the opposite. It might actually cause extinction but we have no idea how likely that is and neither do we have any idea how likely any bad not-quite-extinction outcome is. The outcome might even be very positive. We have no idea when anything will happen and the only realistic plan that's sure to avoid the bad outcome is to stop building AI, which also means we don't get the potential good outcome, and there's no scientific consensus about that (or anything else) being a good plan because it's almost impossible to gather concrete empirical evidence about the risk. By the time such evidence is available, it might be too late (this could also have happened with climate change, we got lucky there...)
That is the question, though I'd turn it around on you - over the course of human history, the speed of progress has been ever-increasing. As AI develops, it is itself a new tool that should increase the speed of progress. Shouldn't our base case be the assumption that progress will speed up, rather than to question whether it's maintainable?
Actually, I could see that happening.
Maybe it's time to give AGI a chance to run things anyway and see if it can do any better. Certainly it isn't a very high bar.
• driving (at human level safety)
• folding clothes with two robotic hands
• write mostly correct code at large scale (not just leetcode problems), fix bugs after testing
• ability to reason beyond simple riddles
• perform simple surgeries unassisted
• look at a recipe and cook a meal
• most importantly, ability to learn new skills at average human level. Ability to figure out what it needs to learn to solve a given problem, watch some tutorials, and learn from that.
> Neurons do not work alone. Instead, they depend heavily on non-neuronal or “glia” cells for many important services including access to nutrition and oxygen, waste clearance, and regulation of the ions such as calcium that help them build up or disperse electric charge.
That's exactly what homeostatisis is but we don't simulate astrocyte mitochondria to understand what effect they have on another neuron's activation. They are independent. Otherwise, biochemistry wouldn't function at all.
It's also been overestimated tons of times. Look at some of the predictions from the past. It's been a complete crap-shoot. Many things have changed significantly less than people have predicted, or in significantly different ways, or significantly more.
Just because things are accelerating great pace right now doesn't really mean anything for the future. Look at the predictions people made during the "space age" 1950s and 60s. A well-known example would be 2001 (the film and novel). Yes, it's "just" some fiction, but it was also a serious attempt at predicting what the future would roughly look like, and Arthur C. Clarke wasn't some dumb yahoo either.
The year 2001 is more than 20 years in the past, and obviously we're nowhere near the world of 2001, for various reasons. Other examples include things like the Von Braun wheel, predictions from serious scientists that we'd have a moon colony by the 1990s, etc. etc. There were tons of predictions and almost none of them have come true.
They all assumed that the rate of progress would continue as it had, but it didn't, for technical, economical, and pragmatic reasons. What's the point of establishing an expensive moon colony when we've got a perfectly functional planet right here? Air is nice (in spite of what Spongebob says). Plants are nice. Water is nice. Non-cramped space to live in is nice. A magnetosphere to protect us from radiation is nice. We kind of need these things to survive and none are present on the moon.
Even when people are right they're wrong. See "Arthur C Clarke predicts the internet in 1964"[1]. He did accurately predict the internet; "a man could conduct his business just as well from Bali as London" pretty much predicts all the "digital nomads" in Bali today, right?
But he also predicts that the city will be obsolete and "seizes to make any sense". Clearly that part hasn't come true, and likely never will. Can't "remotely" get a haircut, or get a pint with friends, or all sorts of other things. And where are all those remote workers in Bali? In the Denpasar/Kuta/Canggu area. That is: a city.
It's half right and half wrong.
The take-away is that predicting the future is hard, and that anyone who claims to predicts the future with great certainty is a bullshitter, idiot, or both.
I'm not saying I agree, I'm not really sure how useful it is as a term, seems to me any definition would be arbitrary - we'll always want more intelligence, it doesn't really matter if it's reached a level we can call 'general' or not.
(More useful in specialised roles perhaps, like the 'levels' of self-driving capability.)
> they showed in live, behaving animals that they could enhance the response of visual cortex neurons to visual stimulation by directly controlling the activity of astrocytes.
Perhaps we're talking past each other, but I thought you were implying that since some function supports homeostasis, we can assume it doesn't matter to a larger computation, and don't need to model it. That's not true with astrocytes, and I wouldn't be surprised if we eventually find out that other biological functions (like "junk DNA") fall into that category as well.
I was only referring to the internal processes of a cell. We don't need to simulate 90+% of the biochemical processes in a neuron to get an accurate simulation of that neuron - if we did it'd pretty much fuck up our understanding of every other cell because most cells share the same metabolic machinery.
The characteristics of the larger network and which cells are involved is an open question in neuroscience and it's largely an intractable problem as of this time.
Just off the top of my head, in my lifetime, I have seen discoveries regarding new neuropeptides/neurotransmitters such as orexin, starting to understand glial cells, new treatments for brain diseases such as epilepsy, new insight into neural metabolism, and better mapping of human neuroanatomy. I might only be a layman observing, but I have a hard time believing anyone can think we've made almost no progress.
[0] https://en.wikipedia.org/wiki/History_of_artificial_neural_n...
Francis Fukuyama wrote in "The Last Man":
> The life of the last man is one of physical security and material plenty, precisely what Western politicians are fond of promising their electorates. Is this really what the human story has been "all about" these past few millennia? Should we fear that we will be both happy and satisfied with our situation, no longer human beings but animals of the genus homo sapiens?
It's a fantastic essay (really, the second half of his seminal book) that I think everyone should read
I think this is the big difference between what you're describing and AI. AI already exists, unlike a moon colony, so we're talking about pushing something forward vs. creating brand new things. It's also pretty well established that it's got tremendous economic value, which means that in our capitalist society, it's going to have a lot of resources directed at it. Not necessarily the case for a moon colony whose economic value is speculative and much longer term.
Then those AIs aren't generally intelligences, as you said they are specialized.
Note that a set of AIs is still an AI, so AI should always be compared to groups of humans and not a single human. Since the AI needs to replace groups of humans and not individuals, very few workplaces has individual humans doing tasks alone without talking to coworkers.
That was really my point: you can't really predict what the future will bring based on what we can do today. People were extrapolating from "we've got fancy rockets and space satellites" to "moon base" in the past, and now they're extrapolating from "GPT-4" to "GPT-5 will replace $thing soon" and even "major step towards AGI". I don't think you can make that assumption.
I'm also somewhat skeptical on the economic value, but that's a long argument I don't have time to expand on right now, and this margin is too narrow to contain it.
Happiness is always fleeting. Aren't our lives a bit dystopian already if we need to do work and for what reason? So that we can possibly feel like we are meaningful with hopes that we don't lose our ability to be useful.
Idle curiosity, but what NLP tools evaluate translation quality better than a person? I was under the (perhaps mistaken) impression that NLP tools would be designed to approximate human intuition on this.
Their results are also highly biased. Most senior researchers aren't going to waste their time filling this out (90% of people did not fill it out). They almost certainly got very junior people and those with an axe to grind. Many of the respondents also have a conflict of interest, they run AI startups.
The survey does address the points above a bit. Per Section 5.2.2 and Appendix D, the survey had a response rate of 15% overall and of ~10% among people with over 1000 citations. Respondents who had given "when HLMI [more or less AGI] will be developed" or "impacts of smarter-than-human machines" a "great deal" of thought prior to the survey were 7.6% and 10.3%, respectively. Appendix D indicates that they saw no large differences between industry and academic respondents besides response rate, which was much lower for people in industry.
This is a long story. But, your take on this question is what the average person who responded to that survey knows. And it shows you really how little the results mean. Here are some minutia that really matter:
1. Even if you measure quality with people in the loop. What do you ask people? Here's a passage in English, one in French, do you agree? Rate it out of 10? Turns out people aren't calibrated at all to give reasonable ratings, you get basically junk results if you run this experiment.
2. You can ask people to do head to head experiments. Do you like translation A more than translation B? But.. what criteria should they use? Is accuracy what matters most? Is it how would they translate? Is it how well A or B reads? Is it how well it represents the form of the source? Or the ideas of the source?
3. Are we measuring sentences? Paragraphs? Pages? 3 word sentences "give me grool" are pretty easy. 3 page translations get tricky. Now you want to represent something about the style of the writer. Or to realize that they're holding something back. For example, it can be really obvious in French that I'm holding back someone's gender, but not obvious at all in English. What about customs? Taboos? Do we even measure 3 pages worth of translation in our NLP corpora? The respondents have no idea.
4. There are even domain-specific questions about translations. Do you know how to evaluate English to French in the context of a contract? One that goes from common law to civil law? No way. You need to translate ideas now, not just words. How about medical translation? Most translation work is highly technical like this.
I could go on. Mostly we don't even measure minutia about translations or domain-specific translation in our NLP benchmarks because the tools aren't good enough for that. Nor do we measure 5 page translations for their fidelity.
We actually mostly don't measure translations using humans at all! We collect translations from humans and then we compare machine translations to human translations after the fact, with something called parallel corpora (the historical example is the Hansard corpus; which is the proceedings of the Canadian parliament that are manually translated in English and French; the EU has also been a boon for translation research).
I'm scratching the surface here. Translation is a really complicated topic. My favourite book related to this is the Dictionary of Untranslatables https://press.princeton.edu/books/hardcover/9780691138701/di... Not something you'd read end-to-end but a really fun reference to dip into once in a while.
If someone who knows about these issues wants to say that there will be human-level translation AI in 10 years, ok, fine I'm willing to buy that. But if someone who is ignorant of all of this is trying to tell me that there will be human level AI for translation in 10 years, eh, they just don't know what they're talking about. I am by the way a translation visitor, I've published in the area, but I'm not an expert at all, I don't even trust my opinion on the subject of when it will be automated.
About biases, I saw appendix A and D.
Seniority doesn't mean >1000 citations. There are master's students with 1000 citations in junk journals who happened to get a paper in a better venue. Number of citations is not an indication of anything.
The way they count academia vs industry is meaningless. There are plenty of people who have an affiliation to a university but are primarily at a startup. There are plenty of people who are minor coauthors on a paper, or even faculty who are mostly interested in making money off of the AI hype. There are plenty of people who graduated 3 years ago, this is a wrap-up of their work, they counted as academic in the survey, but now they're in industry. etc.
- Go on Linkedin or fiverr and look at the kinds of jobs being offered remote right now. developer, HR, bureaucrat, therapeut, editor, artist etc. Current AI agents can not do the large majority of these jobs just like that, without supervision. Yes they can perform certain aspects of the job, but not the actual job, people wouldn't hire them.
A hard Turing test is a proper Turing test that's long and not just smalltalk. Intelligence can't be "faked" then. Even harder is when it is performed adversarially, i.e. there is a team of humans that plans which questions it will ask and really digs deep. For example: commonsense reasoning and long-term memory are two pureky textual tasks where LLMs still fail. Yes they do amazingly well in comparison go what we had previously, which was nothing, but if you think they are human equivalent then imo you need to play with LLMs more.
Another hard Turing test would be: Can this agent be a fulfilling long-distance partner? And I'm not talking about fulfilling like current people are having relationships with crude agents. I am talking about really giving you the sense of being understood, learning you, enriching your live etc. We can't do that yet.
Give me an agent and 1 week and I can absolutely figure out whether it is a human or AI.