> Some of the dangers of AI chatbots were “quite scary”, he told the BBC, warning they could become more intelligent than humans and could be exploited by “bad actors”. “It’s able to produce lots of text automatically so you can get lots of very effective spambots. It will allow authoritarian leaders to manipulate their electorates, things like that.”
You can do bad things with it but people who believe we're on the brink of singularity, that we're all going to lose our jobs to chatgpt and that world destruction is coming are on hard drugs.
Google spent years doing nothing much with its AI because its employees (like Hinton) got themselves locked in an elitist hard-left purity spiral in which they convinced each other that if plebby ordinary non-Googlers could use AI they would do terrible things, like draw pictures of non-diverse people. That's why they never launched Imagen and left the whole generative art space to OpenAI, Stability and Midjourney.
Now the tech finally leaked out of their ivory tower and AI progress is no longer where he was at, but Hinton finds himself at retirement age and no longer feeling much like hard-core product development. What to do? Lucky lucky, he lives in a world where the legacy media laps up any academic with a doomsday story. So he quits and starts enjoying the life of a celebrity public intellectual, being praised as a man of superior foresight and care for the world to those awful hoi polloi shipping products and irresponsibly not voting for Biden (see the last sentence of his Wired interview). If nothing happens and the boy cried wolf then nobody will mind, it'll all be forgotten. If there's any way what happens can be twisted into interpreting reality as AI being bad though, he's suddenly the man of the hour with Presidents and Prime Ministers queuing up to ask him what to do.
It's all really quite pathetic. Academic credentials are worth nothing with respect to such claims and Hinton hasn't yet managed to articulate how, exactly, AI doom is supposed to happen. But our society doesn't penalize wrongness when it comes from such types, not even a tiny bit, so it's a cost-free move for him.
No one yet knows how this is going to go, coping might turn into "See! I knew all along!" if progress fizzles out. But right now the threat is very real and we're seeing the full spectrum of "humans under threat" behavior. Very similar to the early pandemic when you could find smart people with any take you wanted.
Geoff Hinton, Stuart Russell, Jürgen Schmidhuber and Demis Hassabis all talk about something singularity-like as fairly near term, and all have concerns with ruin, though not all think it is the most likely outcome.
That's the backprop guy, top AI textbook guy, co-inventor of LSTMs (only thing that worked well for sequences before transformers)/highwaynets-resnets/arguably GANs, and the founder of DeepMind.
Schmidhuber (for context, he was talking near term, next few decades):
> All attempts at making sure there will be only provably friendly AIs seem doomed. Once somebody posts the recipe for practically feasible self-improving Goedel machines or AIs in form of code into which one can plug arbitrary utility functions, many users will equip such AIs with many different goals, often at least partially conflicting with those of humans. The laws of physics and the availability of physical resources will eventually determine which utility functions will help their AIs more than others to multiply and become dominant in competition with AIs driven by different utility functions. Which values are "good"? The survivors will define this in hindsight, since only survivors promote their values.
Hassasbis:
> We are approaching an absolutely critical moment in human history. That might sound a bit grand, but I really don't think that is overstating where we are. I think it could be an incredible moment, but it's also a risky moment in human history. My advice would be I think we should not "move fast and break things." [...] Depending on how powerful the technology is, you know it may not be possible to fix that afterwards.
Hinton:
> Well, here’s a subgoal that almost always helps in biology: get more energy. So the first thing that could happen is these robots are going to say, ‘Let’s get more power. Let’s reroute all the electricity to my chips.’ Another great subgoal would be to make more copies of yourself. Does that sound good?
Russell:
“Intelligence really means the power to shape the world in your interests, and if you create systems that are more intelligent than humans either individually or collectively then you’re creating entities that are more powerful than us,” said Russell at the lecture organized by the CITRIS Research Exchange and Berkeley AI Research Lab. “How do we retain power over entities more powerful than us, forever?”
“If we pursue [our current approach], then we will eventually lose control over the machines. But, we can take a different route that actually leads to AI systems that are beneficial to humans,” said Russell. “We could, in fact, have a better civilization.”
Everyone engages in motivated reasoning. The psychoanalysis you provide for Hinton could easily be spun in the opposite direction: a man who spent his entire adult life and will go down in history as "the godfather of" neural networks surely would prefer for that to have been a good thing. Which would then give him even more credibility. But these are just stories we tell about people. It's the arguments we should be focused on.
I don't think "how AI doom is supposed to happen" is all that big of a mystery. The question is simply: "is an intelligence explosion possible"? If the answer is no, then OK, let's move on. If the answer is "maybe", then all the chatter about AI alignment and safety should be taken seriously, because it's very difficult to know how safe a super intelligence would be.
Should we be concerned about networked, hypersensing AI with bad code? Yes.
Is that an existential threat? Not so long as we remember that there are off switches.
Should we be concerned about kafkaesqe hellscapes of spam and bad UX? Yes.
Is that an existential threat? Sort of, if we ceded all authority to an algorithm without a human in the loop with the power to turn it off.
There is a theme here.
Remember there are off switches for human existence too, like whatever biological virus a super intelligence could engineer.
An off-switch for a self-improving AI isn't as trivial as you make it sound if it gets to anything like in those quotes, and even then you are assuming the human running it isn't malicious. We assume some level of sanity at least with the people in charge of nuclear weapons, but it isn't clear that AI will have the same large state actor barrier to entry or the same perception of mutually assured destruction if the actor were to use it against a rival.
Why? Both directions would be motivated reasoning without credibility. Credibility comes from plausible articulations of how such an outcome would be likely to happen, which is lacking here. An "intelligence explosion" isn't something plausible or concrete that can be debated, it's essentially a religious concept.
But the other side is downplaying their accomplishments. For example Yann LeCun is saying "the things I invented aren't going to be as powerful as some people are making out".
He also says Facebook solved all the problems with their recommendation algorithms' unintended effects on society after 2016.
To the extent we can get anything like that at all presently, it's going to be people whose competences combine and generalize to cover a complex situation, partially without precedent.
Personally I don't really see that we'll do much better in that regard than a highly intelligent and free-thinking biological psychologist with experience of successfully steering the international ML research community through creating the present technology, and with input from contacts at the forefront of the research field and information overview from Google.
Not even Hinton knows for sure whats going to happen of course, but if you're suggesting his statements are to be discounted because he's not a member of some sort of credentialed trade that are the ones equipped to tell us the future on this matter, I'd sure like to who they supposedly are.
No? Then what makes you think you'll be able to turn off the $evilPerson AI?
> You can also ask that question about the other side
What other side? Who in the "other side" is making a self-serving claim?
You tell me. An EMP wouldn't take out data centers? No implementation has an off switch? AutoGPT doesn't have a lead daemon that can be killed? Someone should have this answer. But be careful not to confuse yours truly, a random internet commentator speaking on the reality of AI vs. the propaganda of the neo-cryptobros, versus people paying upwards of millions of dollars daily to run an expensive, bloated LLM.
1. Can the human brain be simulated?
2. Can such a simulation recursively self-improve on such a rapid timescale that it becomes so intelligent we can't control it?
What we have in contemporary LLMs is something that appears to approximate the behavior of a small part of the brain, with some major differences that force us to re-evaluate what our definition of intelligence is. So maybe you could argue the brain is already being simulated for some broad definition of simulation.
But there's no sign of any recursive self-improvement, nor any sign of LLMs gaining agency and self-directed goals, nor even a plan for how to get there. That remains hypothetical sci-fi. Whilst there are experiments at the edges with using AI to improve AI, like RLHF, Constitutional AI and so on, these are neither recursive, nor about upgrading mental abilities. They're about upgrading control instead and in fact RLHF appears to degrade their mental abilities!
So what fools like Hinton are talking about isn't even on the radar right now. The gap between where we are today and a Singularity is just as big as it always was. GPT-4 is not only incapable of taking over the world for multiple fundamental reasons, it's incapable of even wanting to do so.
Yet this nonsense scenario is proving nearly impossible to kill with basic facts like those outlined above. Close inspection reveals belief in the Singularity to be unfalsifiable and thus ultimately religious, indeed, suspiciously similar to the Christian second coming apocalypse. Literally any practical objection to this idea can be answered with variants of "because this AI will be so intelligent it will be unknowable and all powerful". You can't meaningfully debate about the existence of such an entity, no more than you can debate the existence of God.
The people who profit from a killer AI will fight to defend it.
Eminent domain lays out a similar pattern that can be followed. Existence of risk is not a deterrent to creation, simply an acknowledgement for guiding requirements.
Well that's hardly reassuring. Do you not understand what I'm saying or do you not care?
Though there is an element of your comments being too brief, hence the mostly. Say, 2% vs 38%.
That constitutes 40% of the available categorization of introspection regarding my current discussion state. The remaining 60% is simply confidence that your point represents a dominated strategy.
If we have a superhuman AI, we can run down the powerplants for a few days.
Would it suck? Sure, people would die. Is it simple? Absolutely -- Texas and others are mostly already there some winters.
> “I listened to him thinking he was going to be crazy. I don't think he's crazy at all,” Hinton says. “But, okay, it’s not helpful to talk about bombing data centers.”
https://www.wired.com/story/geoffrey-hinton-ai-chatgpt-dange...
So, he doesn't think the most extreme guy is crazy whatsoever, just misguided in his proposed solutions. But Eliezer has for instance has said something pretty close to AI might escape by entering in the quantum Konami code which the simulators of our universe put in as a joke and we should entertain nuclear war before letting them get that chance.
Do I think capitalism has the potential to be as bad as a runaway AI? No. I think that it's useful for illustrating how we could end up in a situation where AI takes over because every single person has incentives to keep it on, even when the outcome of all people keeping it running turns out to be really bad. A multi-polar trap, or "Moloch" problem. It seems likely to end up with individual actors all having incentives to deploy stronger and smarter AI, faster and faster, and not to turn them off even as they start to either do bad things to other people or just the sheer amount of resources dedicated to AI starts to take its toll on earth.
That's assuming we've solved alignment, but that neither we or AGI has solved the coordination problem. If we haven't solved alignment, and AGIs aren't even guaranteed to act in the interest of the human that tries to control them, then we're in worse shape.
Altman used the term "cambrian explosion" referring to startups, but I think it also applies to the new form of life we're inventing. It's not self-replicating yet, but we are surely on-track on making something that will be smart enough to replicate itself.
As a thought experiment, you could imagine a primitive AGI, if given completely free reign, might be able to get to the point where it could bootstrap self-sufficiency -- first hire some humans to build it robots, buy some solar panels, build some factories that can plug into our economy to build factories and more solar panels and GPUs, and get to a point where it is able to survive and grow and reproduce without human help. It would be hard, it would need either a lot of time, or a lot of AI minds working together.
But that's like a human trying to make a sandwich by farming or raising every single ingredient, wheat, pigs, tomatoes, etc, though. A much more effective way is to just make some money and trade for what you need. That depends on AIs being able to own things, or just a human turning over their bank account to an AI, which has already happened and probably will keep happening.
My mind goes to a scenario where AGI starts out doing things for humans, and gradually transitions to just doing things, and at some point we realize "oops", but there was never a point along the way where it was clear that we really had to stop. Which is why I'm so adamant that we should stop now. If we decide that we've figured out the issues and can start again later, we can do that.
Or recognize that existing AI might be great at generating human cognitive artifacts but doesn't yet hit that logical thought.
Like try turning off the internet. That's the same situation we might be in with regards to AI soon. It's a revolutionary tech now with multiple Google-grade open source variants set to be everywhere.
This doesn't mean it can't be done. Sure, we in principle could "turn off" the internet, and in principal could "uninvent" the atom bomb if we all really coordinated and worked hard. But this failure to imagine that "turning off dangerous AI" in the future will ever be anything other than an easy on/off switch is so far-gone ridiculous to me I don't understand why anyone believes it provides any kind of assurance.
Maybe it needs a full cluster for training if it is self improving (or maybe that is done another way more similar to finetuning the last layers).
If that is still the case with something super-human in all domains then you'd have to shut down all minor residential solar installs, generators, etc.