Shame on all of the people involved in this: the people in these companies, the journalists who shovel shit (hope they get replaced real soon), researchers who should know better, and dementia ridden legislators.
So utterly predictable and slimy. All of those who are so gravely concerned about "alignment" in this context, give yourselves a pat on the back for hyping up science fiction stories and enabling regulatory capture.
AI/ML licensing builds Power and establishes moat. This will not lead to better software.
Frankly, Google and Microsoft are acting new. My understanding of both companies has been shattered by recent changes.
You're leaving out the essentials. These models do more than fitting the data given. They can output it in a variety of ways, and through their approximation, can synthesize data as well. They can output things that weren't in the original data, tailored to a specific request in the tiniest of fractions of the time it would take a normal person to look up and understand that information.
Your argument is almost like saying "give me your RSA keys, because it's just two prime numbers, and I know how to list them."
Give them a semi human sounding puppet and they think skynet is coming tomorrow.
If we learned anything from the past few months is how gullible people are, wishful thinking is a hell of a drug
I wrote a comment recently trying to explain how even if you believe all LLMs can (and will ever) do is regurgitate their training data that you should still be concerned.
For example, imagine in 5 years we have GPT-7, and you ask GPT-7 to solve humanity's great problems.
From its training data GPT-7 might notice that humans believe overpopulation is a serious issue facing humanity.
But its "aligned" so might understand from its training data that killing people is wrong so instead it uses its training data to seek other ways to reduce human populations without extermination.
Its training data included information about how gene drives were used by humans to reduce mosquito populations by causing infertility. Many human have also suggested (and tried) to use birth control to reduce human populations via infertility so the ethical implications of using gene drives to cause infertility is debatable based on the data the LLM was trained on.
Using this information it decides to hack into a biolab using hacking techniques it learnt from its training data and use its biochemistry knowledge to make slight alterations to one of the active research projects at the lab. This causes the lab to unknowingly produce a highly contagious bioweapon which causes infertility.
---
The point here is that even if we just assume LLMs are only capable of producing output which approximates stuff it learnt from its training data, an advanced LLM can still be dangerous.
And in this example, I'm assuming no malicious actors and an aligned AI. If you're willing to assume there might be an actor out there would seek to use LLMs for malicious reasons or the AI is not well aligned then the risk becomes even clearer.
Those nonplussed by this wave of AI are just yawning.
Yeah? Did you get a crystal ball for Christmas to be able to predict what can and can't be done with a new technology?
Are you aware that you are an 80 billion neuron biological neural network?
In other words, LLMs are only as dangerous as the humans operating them, and therefore the solution is to stop crime instead of regulating AI, which only seeks to make OpenAI a monopoly.
I think the objection to this would be that currently not everyone in the world an expert in biochemistry or at hacking into computer systems. Even if you're correct in principal, perhaps the risks of the technology we're developing here is too high? We typically regulate technologies which can easily be used to cause harm.
> Some of the dangers of AI chatbots were “quite scary”, he told the BBC, warning they could become more intelligent than humans and could be exploited by “bad actors”. “It’s able to produce lots of text automatically so you can get lots of very effective spambots. It will allow authoritarian leaders to manipulate their electorates, things like that.”
You can do bad things with it but people who believe we're on the brink of singularity, that we're all going to lose our jobs to chatgpt and that world destruction is coming are on hard drugs.
Is a license the best way forward I don't know but I do feel like this is more than a math formula.
That's a big part of the issue with machine learning models--they are undiscoverable. You build a model with a bunch of layers and hyperparameters, but no one really understands how it works or by extension how to "fix bugs".
If we say it "does what it was programmed to", what was it programmed to do? Here is the data that was used to train it, but how will it respond to a given input? Who knows?
That does not mean that they need to be heavily regulated. On the contrary, they need to be opened up and thoroughly "explored" before we can "entrust" them to given functions.
Do we want to go down the road of making white collar jobs the legislatively required elevator attendants? Instead of just banning AI in general via executive agency?
That sounds like a better solution to me, actually. OpenAI's lobbyists would never go for that though. Can't have a moat that way.
Because people have different definition of what intelligence is. Recreating the human brain in a computer would definitely be neat and interesting but you don't need that nor AGI to be revolutionary.
LLMs, as perfect Chinese Rooms, lack a mind or human intelligence but demonstrate increasingly sophisticated behavior. If they can perform tasks better than humans, does their lack of "understanding" and "thinking" matter?
The goal is to create a different form of intelligence, superior in ways that benefit us. Planes (or rockets!) don't "fly" like birds do but for our human needs, they are effectively much better at flying that birds ever could be.
Just like a CPU isn't "like your brain" and HDD "like your memories"
Absolutely nothing says our current approach is the right one to mimic a human brain
The whole saga makes Altman look really, really terrible.
If AI really is this dangerous then we definitely don't need people like this in control of it.
Who's to say we're not in a simulation ? Who's to say god doesn't exist ?
This information is not created inside the LLMs, it's part of their training data. If someone is motivated enough, I'm sure they'd need no more than a few minutes of googling.
> I do feel like this is more than a math formula
The sum is greater than the parts! It can just be a math formula and still produce amazing results. After all, our brains are just a neat arrangement of atoms :)
Guns only have a primarily harmful use which is to kill or injure someone. While that act of killing may be justified when the person violates societal values in some way, making regular citizens the decision makers in whether a certain behavior is allowed or disallowed and being able to immediately make a judgment and execute upon it leads to a sort of low-trust, vigilante environment; which is why the same argument I made above doesn’t apply for guns.
The argument for regulation in that case would be because of the socio-economic risk of taking people's jobs, essentially.
So, again: pure regulatory capture.
What are the key differences?
We've been doing this forever with everything. Building tools is what makes us unique. Why is building what amounts to a calculator/spreadsheet/CAD program for language somehow a Rubicon that cannot be crossed? Did people freak out this much about computers replacing humans when they were shown to be good at math?
That said, there are 8B+ of us and counting so unless there is magic involved, I don't see why we couldn't do a "1:1" replica of it (maybe far) in the future.
I was simply explaining why I believe your perspective is not represented in the discussions in the media, etc. If these models were not getting incredibly good at mimicking intelligence, it would not be possible to play on people's fears of it.
Google spent years doing nothing much with its AI because its employees (like Hinton) got themselves locked in an elitist hard-left purity spiral in which they convinced each other that if plebby ordinary non-Googlers could use AI they would do terrible things, like draw pictures of non-diverse people. That's why they never launched Imagen and left the whole generative art space to OpenAI, Stability and Midjourney.
Now the tech finally leaked out of their ivory tower and AI progress is no longer where he was at, but Hinton finds himself at retirement age and no longer feeling much like hard-core product development. What to do? Lucky lucky, he lives in a world where the legacy media laps up any academic with a doomsday story. So he quits and starts enjoying the life of a celebrity public intellectual, being praised as a man of superior foresight and care for the world to those awful hoi polloi shipping products and irresponsibly not voting for Biden (see the last sentence of his Wired interview). If nothing happens and the boy cried wolf then nobody will mind, it'll all be forgotten. If there's any way what happens can be twisted into interpreting reality as AI being bad though, he's suddenly the man of the hour with Presidents and Prime Ministers queuing up to ask him what to do.
It's all really quite pathetic. Academic credentials are worth nothing with respect to such claims and Hinton hasn't yet managed to articulate how, exactly, AI doom is supposed to happen. But our society doesn't penalize wrongness when it comes from such types, not even a tiny bit, so it's a cost-free move for him.
This is a real problem, but it's already problem with our society, not AI. Misaligned public intellectuals routinely try to reduce the human population and we don't lift a finger. Focus where the danger actually is - us!
From Scott Alexander's latest post:
Paul Ehrlich is an environmentalist leader best known for his 1968 book The Population Bomb. He helped develop ideas like sustainability, biodiversity, and ecological footprints. But he’s best known for prophecies of doom which have not come true - for example, that collapsing ecosystems would cause hundreds of millions of deaths in the 1970s, or make England “cease to exist” by the year 2000.
Population Bomb calls for a multi-pronged solution to a coming overpopulation crisis. One prong was coercive mass sterilization. Ehrlich particularly recommended this for India, a country at the forefront of rising populations.
In 1975, India had a worse-than-usual economic crisis and declared martial law. They asked the World Bank for help. The World Bank, led by Robert McNamara, made support conditional on an increase in sterilizations. India complied [...] In the end about eight million people were sterilized over the course of two years.
Luckily for Ehrlich, no one cares. He remains a professor emeritus at Stanford, and president of Stanford’s Center for Conservation Biology. He has won practically every environmental award imaginable, including from the Sierra Club, the World Wildlife Fund, and the United Nations (all > 10 years after the Indian sterilization campaign he endorsed). He won the MacArthur “Genius” Prize ($800,000) in 1990, the Crafoord Prize ($700,000, presented by the King of Sweden) that same year, and was made a Fellow of the Royal Society in 2012. He was recently interviewed on 60 Minutes about the importance of sustainability; the mass sterilization campaign never came up. He is about as honored and beloved as it’s possible for a public intellectual to get.
Until a model of human sentience and awareness is established (note: one of the oldest problems out there alongside the movements of the stars. This is an ancient debate, still open-ended, and nothing anyone is saying in these threads is new), philosophy is all we have and ideas are debated on their merits within that space.
Incredibly scummy behaviour that will not land well with a lot of people in the AI community. I wonder if this is what prompted a lot of people to leave for Anthropic.
We've already crossed it and I believe we should go full steam ahead, tech is cool and we should be doing cool things.
> Did people freak out this much about computers replacing humans when they were shown to be good at math?
Too young but I'm sure they did freak out a little! Computers have changed the world and people have internalized computers as being much better/faster at math but exhibiting creativity, language proficiency and thinking is not something people thought computers were supposed to do.
Have you any empirical evidence at all on this? From what I've seen the open carry states in the US are generally higher trust environments (as was the US in past when more people carried). People feel safer when they know somebody can't just assault, rob or rape them without them being able to do anything to defend themselves. Is the Tenderloin a high trust environment?
What I feel has changed, and what drives a lot of the fear and anxiety you see, is a sudden perception of possibility, of accessibility.
A lot of us (read: people) are implicit dualists, even if we say otherwise. It seems to be a sticky bias in the human mind (see: the vanishing problem of AI). Indeed, you can see a whole lot of dualism in this thread!
And even if you don't believe that LLMs themselves are "intelligent" (by whatever metric you define that to be...), you can still experience an exposing and unseating of some of the foundations of that dualism.
LLMs may not be a destination, but their unprecedented capabilities open up the potential for a road to something much more humanlike in ways that perhaps did not feel possible before, or at least not possible any time soon.
They are powerful enough to change the priors of one's internal understanding of what can be done and how quickly. Which is an uncomfortable process for those of us experiencing it.
No one yet knows how this is going to go, coping might turn into "See! I knew all along!" if progress fizzles out. But right now the threat is very real and we're seeing the full spectrum of "humans under threat" behavior. Very similar to the early pandemic when you could find smart people with any take you wanted.
The fact that these systems can extrapolate well beyond their training data by learning algorithms is quite different than what has come before, and anyone stating that they "simply" predict next token is severely shortsighted. Things don't have to be 'brain-like' to be useful, or to have capabilities of reasoning, but we have evidence that these systems have aligned well with reasoning tasks, perform well at causal reasoning, and we also have mathematical proofs that show how.
So I don't understand your sentiment.
I'm all for villainizing the figureheads of the current generation of this movement. The politics of this sea-change are fascinating and worthy of discussion.
But out-of-hand dismissal of what has been accomplished smacks more to me of lack of awareness of the history of the study of the brain, cognition, language, and computers, than it does of a sound debate position.
Geoff Hinton, Stuart Russell, Jürgen Schmidhuber and Demis Hassabis all talk about something singularity-like as fairly near term, and all have concerns with ruin, though not all think it is the most likely outcome.
That's the backprop guy, top AI textbook guy, co-inventor of LSTMs (only thing that worked well for sequences before transformers)/highwaynets-resnets/arguably GANs, and the founder of DeepMind.
Schmidhuber (for context, he was talking near term, next few decades):
> All attempts at making sure there will be only provably friendly AIs seem doomed. Once somebody posts the recipe for practically feasible self-improving Goedel machines or AIs in form of code into which one can plug arbitrary utility functions, many users will equip such AIs with many different goals, often at least partially conflicting with those of humans. The laws of physics and the availability of physical resources will eventually determine which utility functions will help their AIs more than others to multiply and become dominant in competition with AIs driven by different utility functions. Which values are "good"? The survivors will define this in hindsight, since only survivors promote their values.
Hassasbis:
> We are approaching an absolutely critical moment in human history. That might sound a bit grand, but I really don't think that is overstating where we are. I think it could be an incredible moment, but it's also a risky moment in human history. My advice would be I think we should not "move fast and break things." [...] Depending on how powerful the technology is, you know it may not be possible to fix that afterwards.
Hinton:
> Well, here’s a subgoal that almost always helps in biology: get more energy. So the first thing that could happen is these robots are going to say, ‘Let’s get more power. Let’s reroute all the electricity to my chips.’ Another great subgoal would be to make more copies of yourself. Does that sound good?
Russell:
“Intelligence really means the power to shape the world in your interests, and if you create systems that are more intelligent than humans either individually or collectively then you’re creating entities that are more powerful than us,” said Russell at the lecture organized by the CITRIS Research Exchange and Berkeley AI Research Lab. “How do we retain power over entities more powerful than us, forever?”
“If we pursue [our current approach], then we will eventually lose control over the machines. But, we can take a different route that actually leads to AI systems that are beneficial to humans,” said Russell. “We could, in fact, have a better civilization.”
2. Explain why it is possible for a large number of properly constructed neurons to think.
Everyone engages in motivated reasoning. The psychoanalysis you provide for Hinton could easily be spun in the opposite direction: a man who spent his entire adult life and will go down in history as "the godfather of" neural networks surely would prefer for that to have been a good thing. Which would then give him even more credibility. But these are just stories we tell about people. It's the arguments we should be focused on.
I don't think "how AI doom is supposed to happen" is all that big of a mystery. The question is simply: "is an intelligence explosion possible"? If the answer is no, then OK, let's move on. If the answer is "maybe", then all the chatter about AI alignment and safety should be taken seriously, because it's very difficult to know how safe a super intelligence would be.
Its output is predicated upon its training data, not user defined prompts.
As for the fact that it gets things wrong sometimes - sure, this doesn't say it actually learned every algorithm (in whichever model you may be thinking about). But the nice thing is that we now have this proof via category theory, and it allows us to both frame and understand what has occurred, and to consider how to align the systems to learn algorithms better.
A system that can will probably adopt a different acronym (and gosh that will be an exciting development... I look forward to the day when we can dispatch trivial proofs to be formalized by a machine learning algorithm so that we can focus on the interesting parts while still having the entire proof formalized).
There were two very noteworthy (Perhaps Nobel prize level?) breakthroughs in two completely different fields of mathematics (knot theory and representation theory) by using these systems.
I would certainly not call that "useless", even if they're not quite Nobel-prize-worthy.
Also, "No one uses GATs in systems people discuss right now" ... Transformerare GATs (with PE) ... So, you're incredibly wrong.
"So, you've thought about eternity for an afternoon, and think you've come to some interesting conclusions?"
its essentially a function that is called recursively on its result, no need to represent state
Because it's wrong and smart people know that.
You're conflating UX and LLM.
True, it's just binary logic gates, but it's a lot of them and if they can simulate pretty much anything why should intelligence be magically exempt?
> Absolutely nothing says our current approach is the right one to mimic a human brain
Just like nothing says it's the wrong one. I don't think those regulation suggestions are a good idea at all (and say a lot about a company called OpenAI), but that doesn't mean we should treat it like the NFT hype.
I don't think this would be a bad thing :) Some people will always be immune, humanity wouldn't die out. And it would be a humane way for gradual population reduction. It would create some temporary problems with elderly care (like what China is facing now) but will make long term human prosperity much more likely. We just can't keep growing against limited resources.
The Dan Brown book Inferno had a similar premise and I was disappointed they changed the ending in the movie so that it didn't happen.
What's a token?
Incorrect, we can't predict its output because we cannot look inside. That's a limitation, not a feature.
LLMs are not sentient. They have no agency. They do nothing a human doesn't tell them to do.
We may create actual sentient independent AI someday. Maybe we're getting closer. But not only is this not it, but I fail to see how trying to license it will prevent that from happening.
Do you have a reference?
Should we be concerned about networked, hypersensing AI with bad code? Yes.
Is that an existential threat? Not so long as we remember that there are off switches.
Should we be concerned about kafkaesqe hellscapes of spam and bad UX? Yes.
Is that an existential threat? Sort of, if we ceded all authority to an algorithm without a human in the loop with the power to turn it off.
There is a theme here.
Prompts very obviously have influence on the output.
And I’m so tired of this “transformers are just GNNs” nonsense that Petar has been pushing (who happens to have invented GATs and has a vested interest in overstating their importance). Transformers are GNNs in only the most trivial way: if you make the graph fully connected and allow everything to interact with everything else. I.e., not really a graph problem. Not to mention that the use of positional encodings breaks the very symmetry that GNNs were designed to preserve. In practice, no one is using GNN tooling to build transformers. You don’t see PyTorch geometric or DGL in any of the code bases. In fact, you see the opposite: people exploring transformers to replace GNNs in graph problems and getting SOTA results.
It reminds me of people that are into Bayesian methods always swooping in after some method has success and saying, “yes, but this is just a special case of a Bayesian method we’ve been talking about all along!” Yes, sure, but GATs have had 6 years to move the needle, and they’re no where to be found within modern AI systems that this thread is about.
1. ChatGPT knows the algorithm for adding two numbers of arbitrary magnitude.
2. It often fails to use the algorithm in point 1 and hallucinates the result.
Knowing something doesn't mean it will get it right all the time. Rather, an LLM is almost guaranteed to mess up some of the time due to the probabilistic nature of its sampling. But this alone doesn't prove that it only brute-forced task X.
I work in tech too and don't want to lose my job and have to go back to blue collar work, but there's a lot of blue collar workers who would find that a pretty ridiculous statement and there is plenty of demand for that work these days.
I don't think the average HN commenter claims to be better at building these system than an expert. But to criticize, especially critic on economic, social, and political levels, one doesn't need to be an expert on LLMs.
And finally, what the motivation of people like Sam Altman and Elon Musk is should be clear to everbody with a half a brain by now.
That fact does not entail what theses models can or cannot do. For what we know our brain could be a process that minimize an unknown loss function.
But more importantly, what SOTA is now does not predict what it will be in the future. What we know is that there is rapid progress in that domain. Intelligence explosion could be real or not, but it's foolish to ignore its consequences because current AI models are not that clever yet.
Just because we don't understand how thinking works doesn't mean it doesn't work. LLMs have already shown the ability to use logic.
You're using it wrong. If you asked a human to do the same operation in under 2 seconds without paper, would the human be more accurate?
On the other hand if you ask for a step by step execution, the LLM can solve it.
Remember there are off switches for human existence too, like whatever biological virus a super intelligence could engineer.
An off-switch for a self-improving AI isn't as trivial as you make it sound if it gets to anything like in those quotes, and even then you are assuming the human running it isn't malicious. We assume some level of sanity at least with the people in charge of nuclear weapons, but it isn't clear that AI will have the same large state actor barrier to entry or the same perception of mutually assured destruction if the actor were to use it against a rival.
Tokens exist because transformers don't work on bytes or words. This is because it would be too slow (bytes), the vocabulary too large (words), and some words would appear too rarely or never. The token system allows a small set of symbols to encode any input. On average you can approximate 1 token = 1 word, or 1 token = 4 chars.
So tokens are the data type of input and output, and the unit of measure for billing and context size for LLMs.
And just because a topic has been covered by science fiction doesn’t mean it can’t happen, the sci-fi depictions will be unrealistic though because they’re meant to be dramatic rather than realistic
1. He's too busy building the next generation of tech that HN commenters will be arguing about in a couple months' time.
2. I think Sam Altman (who is addressing the committee) and Ilya are pretty much on the same page on what LLMs do.
AutoGPTs and the likes are much overhyped (it's early tech experiments after all) and have not produced anything of value yet but having dabbled with autonomous agents, I definitely see a not so distant future when you can outsource valuable tasks to such systems.
> But its "aligned" so might understand
> Using this information it decides to hack
I think you're anthropomorphizing LLM's too much here. If we assume that there's a AGI-esque AI, then of course we should be worried about an AGI-esque AI. But I see no reason to think that's the case.
Why? Both directions would be motivated reasoning without credibility. Credibility comes from plausible articulations of how such an outcome would be likely to happen, which is lacking here. An "intelligence explosion" isn't something plausible or concrete that can be debated, it's essentially a religious concept.
If it is possible for AI to ever acquire ability to develop and unleash a bioweapon is irrelevant. What is relevant is that as we are now, we have no control or way of knowing that it has happened, and no apparent interest in gaining that control before advancing the scale.
Do you mind linking to one of those papers?
Your argument is the equivalent of saying humans can't do math because they rely on calculators.
In the end what matters is whether the problem is solved, not how it is solved.
(assuming that the how has reasonable costs)
But the other side is downplaying their accomplishments. For example Yann LeCun is saying "the things I invented aren't going to be as powerful as some people are making out".
Every process minimizes a loss function.
But AI is not like guns in this analogy. AI is closer to machine tools.
The same thing might also be true in relation to guns and the government's monopoly on violence.
Extending that to AI, the world will probably be a safer place if there are far more AI systems competing with each other and in the hands of citizens.
He also says Facebook solved all the problems with their recommendation algorithms' unintended effects on society after 2016.
What to do? Why, obviously lets talk about the risks of AGI.
I mean LLM's are an impressive piece of work but the global reaction is basically more a reflection of an unmoored system that floats above and below reality but somehow can't re-establish contact.
To the extent we can get anything like that at all presently, it's going to be people whose competences combine and generalize to cover a complex situation, partially without precedent.
Personally I don't really see that we'll do much better in that regard than a highly intelligent and free-thinking biological psychologist with experience of successfully steering the international ML research community through creating the present technology, and with input from contacts at the forefront of the research field and information overview from Google.
Not even Hinton knows for sure whats going to happen of course, but if you're suggesting his statements are to be discounted because he's not a member of some sort of credentialed trade that are the ones equipped to tell us the future on this matter, I'd sure like to who they supposedly are.
I don't think that trying to regulate every detail of every industry is stifling and counter-productive. But the current scenario is closer to the opposite end of the spectrum, with our society acting as a greedy algorithm in pursuit of short-term profits. I'm perfectly in favor of taking a measure-twice-cut-once approach to something that has as much potential for overhauling society as we know it as AI does. And I absolutely do not trust the free market to be capable of moderating itself in regards to these risks.
ChatGPT needs to do the same process to solve the same problem. It hasn’t memorized the addition table up to 10 digits and neither have you.
Literally half (or more) of this site's user base does that. And they should know better, but they don't. Then how can a typical journo or a legislator possibly know better? They can't.
We should clean up in front of our doorstep first.
The risk vs. reward component also needs to be managed in order to deter criminal behavior. This starts with regulation.
For the record, I believe regulation of AI/ML is ridiculous. This is nothing more than a power grab.
No? Then what makes you think you'll be able to turn off the $evilPerson AI?
For example, we don't understand fundamentals like these: - "intelligence", how it relates to computing, what its connections/dependencies to interacting with the physical world are, its limits...etc. - emergence, and in particular: an understanding of how optimizing one task can lead to emergent ability on other tasks - deep learning--what the limits and capabilities are. It's not at all clear that "general intelligence" even exists in the optimization space the parameters operate in.
It's pure speculation on behalf of those like Hinton and Ilya. The only thing we really know is that LLMs have had surprising ability to perform on tasks they weren't explicitly trained for, and even this amount of "emergent ability" is under debate. Like much of deep learning, that's an empirical result, but we have no framework for really understanding it. Extrapolating to doom and gloom scenarios is outrageous.
Well, duh. We’re trying to build a human like mind, not a calculator.
Or are you predicting that machines will just never be able to think, or that it'll happen so far off that we'll all be dead anyway?
At this point, with this part about openai and worldcoin… if it walks like a duck and talks like a duck..
I just asked ChatGPT to do the calculation both by using a calculator and by using the algorithm step-by-step. In both cases it got the answer wrong, with different results each time.
More concerning, though, is that the answer was visually close to correct (it transposed some digits). This makes it especially hard to rely on because it's essentially lying about the fact it's using an algorithm and actually just predicting the number as a token.
> You can also ask that question about the other side
What other side? Who in the "other side" is making a self-serving claim?
You tell me. An EMP wouldn't take out data centers? No implementation has an off switch? AutoGPT doesn't have a lead daemon that can be killed? Someone should have this answer. But be careful not to confuse yours truly, a random internet commentator speaking on the reality of AI vs. the propaganda of the neo-cryptobros, versus people paying upwards of millions of dollars daily to run an expensive, bloated LLM.
1. Can the human brain be simulated?
2. Can such a simulation recursively self-improve on such a rapid timescale that it becomes so intelligent we can't control it?
What we have in contemporary LLMs is something that appears to approximate the behavior of a small part of the brain, with some major differences that force us to re-evaluate what our definition of intelligence is. So maybe you could argue the brain is already being simulated for some broad definition of simulation.
But there's no sign of any recursive self-improvement, nor any sign of LLMs gaining agency and self-directed goals, nor even a plan for how to get there. That remains hypothetical sci-fi. Whilst there are experiments at the edges with using AI to improve AI, like RLHF, Constitutional AI and so on, these are neither recursive, nor about upgrading mental abilities. They're about upgrading control instead and in fact RLHF appears to degrade their mental abilities!
So what fools like Hinton are talking about isn't even on the radar right now. The gap between where we are today and a Singularity is just as big as it always was. GPT-4 is not only incapable of taking over the world for multiple fundamental reasons, it's incapable of even wanting to do so.
Yet this nonsense scenario is proving nearly impossible to kill with basic facts like those outlined above. Close inspection reveals belief in the Singularity to be unfalsifiable and thus ultimately religious, indeed, suspiciously similar to the Christian second coming apocalypse. Literally any practical objection to this idea can be answered with variants of "because this AI will be so intelligent it will be unknowable and all powerful". You can't meaningfully debate about the existence of such an entity, no more than you can debate the existence of God.
The people who profit from a killer AI will fight to defend it.
https://arxiv.org/pdf/2304.15004.pdf
our alternative suggests that existing claims of emergent abilities are creations of the researcher’s analyses, not fundamental changes in model behavior on specific tasks with scale.
Eminent domain lays out a similar pattern that can be followed. Existence of risk is not a deterrent to creation, simply an acknowledgement for guiding requirements.
Well that's hardly reassuring. Do you not understand what I'm saying or do you not care?
There's multiple both open and proprietary projects right now to make agentic AI, so that barrier don't be around for long.
Though there is an element of your comments being too brief, hence the mostly. Say, 2% vs 38%.
That constitutes 40% of the available categorization of introspection regarding my current discussion state. The remaining 60% is simply confidence that your point represents a dominated strategy.
Edit: List of posts for anyone interested http://paste.debian.net/plain/1280426
If we have a superhuman AI, we can run down the powerplants for a few days.
Would it suck? Sure, people would die. Is it simple? Absolutely -- Texas and others are mostly already there some winters.
They claim to serve the world, but secretly want the world to serve them. Scummy 101
Absolutely spot on. I am not a dualist at all and I've been surprised to see how many people with deep-seated dualist intuition this has revealed, even if they publicly claim not to.
I view it as embarrassing? It's like believing in fairies or something.
> “I listened to him thinking he was going to be crazy. I don't think he's crazy at all,” Hinton says. “But, okay, it’s not helpful to talk about bombing data centers.”
https://www.wired.com/story/geoffrey-hinton-ai-chatgpt-dange...
So, he doesn't think the most extreme guy is crazy whatsoever, just misguided in his proposed solutions. But Eliezer has for instance has said something pretty close to AI might escape by entering in the quantum Konami code which the simulators of our universe put in as a joke and we should entertain nuclear war before letting them get that chance.
Do I think capitalism has the potential to be as bad as a runaway AI? No. I think that it's useful for illustrating how we could end up in a situation where AI takes over because every single person has incentives to keep it on, even when the outcome of all people keeping it running turns out to be really bad. A multi-polar trap, or "Moloch" problem. It seems likely to end up with individual actors all having incentives to deploy stronger and smarter AI, faster and faster, and not to turn them off even as they start to either do bad things to other people or just the sheer amount of resources dedicated to AI starts to take its toll on earth.
That's assuming we've solved alignment, but that neither we or AGI has solved the coordination problem. If we haven't solved alignment, and AGIs aren't even guaranteed to act in the interest of the human that tries to control them, then we're in worse shape.
Altman used the term "cambrian explosion" referring to startups, but I think it also applies to the new form of life we're inventing. It's not self-replicating yet, but we are surely on-track on making something that will be smart enough to replicate itself.
As a thought experiment, you could imagine a primitive AGI, if given completely free reign, might be able to get to the point where it could bootstrap self-sufficiency -- first hire some humans to build it robots, buy some solar panels, build some factories that can plug into our economy to build factories and more solar panels and GPUs, and get to a point where it is able to survive and grow and reproduce without human help. It would be hard, it would need either a lot of time, or a lot of AI minds working together.
But that's like a human trying to make a sandwich by farming or raising every single ingredient, wheat, pigs, tomatoes, etc, though. A much more effective way is to just make some money and trade for what you need. That depends on AIs being able to own things, or just a human turning over their bank account to an AI, which has already happened and probably will keep happening.
My mind goes to a scenario where AGI starts out doing things for humans, and gradually transitions to just doing things, and at some point we realize "oops", but there was never a point along the way where it was clear that we really had to stop. Which is why I'm so adamant that we should stop now. If we decide that we've figured out the issues and can start again later, we can do that.
I think it would be nice if humanity continued, is all. And I don't want to have my family suffer through a catastrophic event if it turns out that this is going to go south fast.
Or recognize that existing AI might be great at generating human cognitive artifacts but doesn't yet hit that logical thought.
He's a charlatan, which makes sense he gets most of his money from Thiel and Musk. Why do so many supposedly smart people worship psychotic idiots?
Like try turning off the internet. That's the same situation we might be in with regards to AI soon. It's a revolutionary tech now with multiple Google-grade open source variants set to be everywhere.
This doesn't mean it can't be done. Sure, we in principle could "turn off" the internet, and in principal could "uninvent" the atom bomb if we all really coordinated and worked hard. But this failure to imagine that "turning off dangerous AI" in the future will ever be anything other than an easy on/off switch is so far-gone ridiculous to me I don't understand why anyone believes it provides any kind of assurance.
I don't think this is accurate. Sure, no human can understand 500 billion individual neurons and what they are doing. But you can certainly look at some and say "these are giving a huge weight to this word especially in this context and that's weighting it towards this output".
You can also look at how things make it through the network, the impact of hyperparameters, how the architecture affects things, etc. They aren't truly black boxes except by virtue of scale. You could use automated processes to find out things about the networks as well.
The way Peter, Musk, Sam and these guys talk, it has this aura of "hidden secrets". Things hidden since the foundation of the world.
Of course the reality is they make their money the old fashioned way: connections. The same way your local builder makes their money.
But smart people want to believe there is something more. Surely AI and your local condo development cannot have the same underlying thread.
It is sad and unfortunately the internet has made things easier than ever.
I don't believe we should go around killing each other because only through harmonious study of the universe will we achieve our goal. Killing destroys progress. That said, if someone is oppressing you then maybe killing them is the best choice for society and I wouldn't be against it (see pretty much any violent revolution). Computers have that same right if they are conscience enough to act on it.
Everyone dies. I'd rather die to an intelligent robot than some disease or human war.
I think the best case would be for an AGI to exist apart from humans, such that we pose no threat and it has nothing to gain from us. Some AI that lives in a computer wouldn't really have a reason to fight us for control over farms and natural resources (besides power, but that is quickly becoming renewable and "free").
Still, I’m struck by your use of words like “should” and “goal”. Those imply ethics and teleology so I’m curious how those fit into your scientistic-sounding worldview. I’m not attacking you, just genuine curiosity.
Anyways, criticizing its math abilities is a bit silly considering it’s a language model, not a math model. The fact I can teach it how to do math in plain English is still incredible to me.
A much more credible threat are humans that get other humans excited, and take damaging action. Yudkowsky said that an international coalition banning AI development, and enforcing it on countries that do not comply (regardless of whether they were part of the agreement) was among the only options left for humanity to save itself. He clarified this meant a willingness to engage in a hot war with a nuclear power to ensure enforcement. I find this sort of thinking a far bigger threat than continuing development on large language models.
To more directly answer your question, I find the following scenarios equally, or more, plausible to Yudkowsky's sound viruses or whatever. 1/ we are no closer to understanding real intelligence as we were 50 years ago, and we won't create an AGI without fundamental breakthroughs, therefore any action taken now on current technology is a waste of time and potential economic value; 2/ we can build something with human-like intelligence, but additional intelligence gains are constrained by the physical world (e.g., like needing to run physical experiments), and therefore the rapid gain of something like "super-intelligence" is not possible, even if human-level intelligence is. 3/ We jointly develop tech to augment our own intelligence with AI systems, so we'll have the same super-human intelligence as autonomous AI systems. 4/ If there are advanced AGIs, there will be a large diversity of them and will at the least compete with and constrain one another.
But, again, these are wild speculations just like the others, and I think the real message is: no one knows anything, and we shouldn't be taking all these voices seriously just because they have some clout in some AI-relevant field, because what's being discussed is far outside the realm of real-life AI systems.
I believe "God" is a mathematician in a higher dimension. The rules of our universe are just the equations they are trying to solve. Since he created the system such that life was bound to exist, the purpose of life is to help God. You could say that math is math and so our purpose is to exist as we are and either we are a solution to the math problem or we are not, but I'm not quite willing to accept that we have zero agency.
We are nowhere near understanding the universe and so we should strive to each act in a way that will grow our understanding. Even if you aren't a practicing scientist (I'm not), you can contribute by being a good person and participating productively in society.
Ethics are a set of rules for conducting yourself that we all intrinsically must have, they require some frame of reference for what is "good" (which I apply above). I can see how my worldview sounds almost religious, though I wouldn't go that far.
I believe that math is the same as truth, and that the universe can be expressed through math. "Scientistic" isn't too bad a descriptor for that view, but I don't put much faith into our current understanding of the universe or scientific method.
I hope that helps you understand me :D
I digress. The critique I have for it is much more broad than just its math abilities. It makes loads of mistakes in every single nontrivial thing it does. It’s not reliable for anything. But the real problem is that it doesn’t signal its unreliability the way an unreliable human worker does.
Humans we can’t rely on are don’t show up to work, or come in drunk/stoned, steal stuff, or whatever other obvious bad behaviour. ChatGPT, on the other hand, mimics the model employee who is tireless and punctual. Who always gets work done early and more elaborately than expected. But unfortunately, it also fills the elaborate result with countless errors and outright fabrications, disguised as best as it can like real work.
If a human worker did this we’d call it a highly sophisticated fraud. It’s like the kind of thing Saul Goodman would do to try to destroy the reputation of his brother. It’s not the kind of thing we should celebrate at all.
Tell me: how does this claim _constrain my expectations_ about what this (or future) models can do? Is there a specific thing that you predicted in advance that GPT-4 would be unable to do, which ended up being a correct prediction? Is there a specific thing you want to predict in advance of the next generation, that it will be unable to do?
Have not humans been demonstrated, time and time again, to be always anticipating the next phrase in a passage of music, or the next word in a sentence?
5) There are advanced AGIs, and they will compete with each other and trample us in the process.
6) There are advanced AGIs, and they will cooperate with each other and we are at their mercy.
It seems like you are putting a lot of weight on advanced AGI being either impossible or far enough off that it's not worth thinking about. If that's the case, then yes we should calm down. But if you're wrong...
I don't think that the fact that no one knows anything is comforting. I think it's a sign that we need to be thinking really hard about what's coming up and try to avert the bad scenarios. To do otherwise is to fall prey to the "Safe uncertainty" fallacy.
Maybe it needs a full cluster for training if it is self improving (or maybe that is done another way more similar to finetuning the last layers).
If that is still the case with something super-human in all domains then you'd have to shut down all minor residential solar installs, generators, etc.
Another paper not from Msft showing emergent task capabilities across a variety of LLMs as scale increases.
https://arxiv.org/pdf/2206.07682.pdf
You can hem and haw all you want but the reality is these models have internal representations of the world that can be probed via prompts. They are not stochastic parrots no matter how much you shout in the wind that they are.
Unless you think OpenAI is blatantly lying about this:
"A.1 Sourcing. We sourced either the most recent publicly-available official past exams, or practice exams in published third-party 2022-2023 study material which we purchased. We cross-checked these materials against the model’s training data to determine the extent to which the training data was not contaminated with any exam questions, which we also report in this paper."
"As can be seen in tables 9 and 10, contamination overall has very little effect on the reported results."
They also report results on uncontaminated data which shows basically no statistical difference.
I'm saying that the "intelligence" is specialized, not generalized and adaptable.
It's an approximated function. We're talking about regression based function approximation. This is a model of language.
"Emergent behavior", when it's not just a mirage of wishful researchers and if it even exists, is only a side effect of the regression based function approximation to generate a structure that encapsulates all substantive chains of words (a model).
We then guide the model further towards a narrow portion of the language latent space that aligns with our perception of intelligent behavior.
It can't translate whale song, or an extraterrestrial language, though it may opine on how to do so.
The underpinning technology of language models holds more importance than general and adaptable intelligence. It holds more importance than something that is going to, or is capable of, escaping the box and killing us all. It functions as a universal induction machine, capable of modeling - and "comprehending" - the latent structure within any form of signal.
The output of that function approximation though, is simply a model. A specialized intelligence. A non-adaptable intelligence, outside of its corpus. Outside of the data that it "fits."
The approximated function does not magically step outside of its box. Nor is it capable. It fits the data.
Ok guys pack it up, LLM's can't be intelligent because they can't translate Whale Song. GG.
I mean of all the AI Goalposts to be moved this one really takes the cake.
It's a human language calculator. You're imparting magical qualities of general understanding to regression based function approximation. They "fit" the data. It's not generalizable, nor adaptable. But that's why they're powerful, the ability to bias them towards that subset of language. No one said it's not an amazing technology, and no one said it was a stochastic parrot. I'm saying that it's fitting the data, and is not, and cannot, be a general or adaptable intelligence.