zlacker

>>Jimmc4+(OP)
who is this and why is it important? [1]

super-alignment co-lead with Ilya (who resigned yesterday)

what is super alignment? [2]

> We need scientific and technical breakthroughs to steer and control AI systems much smarter than us. Our goal is to solve the core technical challenges of superintelligence alignment by 2027.

[1] https://jan.leike.name/ [2] https://openai.com/superalignment/

>>kamika+t51
My honest-to-god guess is that it just seemed like a needless cost center in a growing business, so there was pressure against them doing the work they wanted to do.

I'm guessing, but OpenAI probably wants to start monetizing, and doesn't feel like they are going to hit a superintelligence, not really. That may have been the goal originally.

>>jvande+n71
>it just seemed like a needless cost center in a growing business

To some of us, that sounds like, "Fire all the climate scientists because they are needless cost center distracting us from the noble goal of burning as much fossil fuel as possible."

>>holler+cb1
It's more like you started a company to research new fuel sources and hired climate scientists to evaluate the environmental impact of those new fuel sources, but during the course of the research you invented a new internal combustion engine that operates 1000% more efficiently so you pivoted your entire business toward that, removing the need for climate scientists.

This is a tortured analogy, but what I'm getting at is, if OpenAI is no longer pursuing AGI/superintelligence, it doesn't need an expensive superintelligence alignment team.

>>mjr00+Uc1
You're much more confident than I am that the researchers at OpenAI (or anyone else currently alive) are masters of their craft to such an extent that they would even be able to predict whether the next big training run they do will result in a superintelligence or not. Another way of saying the same thing is to say that the only way anyone knows that GPT-4 is not dangerously capable is that it has been deployed extensively enough by now that if it was going to harm us, it would've done so by now: not even the researchers that designed and coded-up GPT-4 or watched it during training could predict with any confidence how capable it would be. For example, everyone was quite surprised by its being able to score in the 90th decile on a bar exam.

Also, even if they never produce a superintelligence, they are likely to produce insights that would make it easier for other teams to produce a superintelligence. (Since employees are free to leave OpenAI and join some other team, there is no practical way to prevent the flow of insights out of OpenAI.)

>>holler+Eh1
Call me uninformed, but I do not see a way forward where a statistical model trained to recognise relationships between words or groups of words, and has a front end coded to query that model could suddenly develop its own independence. That's a whole other thing, where the code to interact with it must allow for constant feedback loops of self.improvement and the vast amount of evolutionary activity that entails.

An interactive mathematical model is not going to run away on its own without some very deliberate steps to take it in that direction.

>>anakai+0t3
We have no idea how consciousness works. Just because you dont see a way forward doesnt mean its not there

>>HDThor+vA3
I think the point was that on a purely technical level, the LLMs as currently used can’t do anything on their own. They only continue a prompt when given. It’s not like a LLM could “decide” to hack the NSA and publish the data tomorrow, because it determined that this would help humanity. The only thing it can do is try to make people do something when they read the responses.

>>echoan+LW5
This is a good interpretation of the point I was getting at, yes.