https://openai.com/index/introducing-superalignment/
"Our goal is to solve the core technical challenges of superintelligence alignment in four years.
While this is an incredibly ambitious goal and we’re not guaranteed to succeed, we are optimistic that a focused, concerted effort can solve this problem:C There are many ideas that have shown promise in preliminary experiments, we have increasingly useful metrics for progress, and we can use today’s models to study many of these problems empirically.
Ilya Sutskever (cofounder and Chief Scientist of OpenAI) has made this his core research focus, and will be co-leading the team with Jan Leike (Head of Alignment). Joining the team are researchers and engineers from our previous alignment team, as well as researchers from other teams across the company."
super-alignment co-lead with Ilya (who resigned yesterday)
what is super alignment? [2]
> We need scientific and technical breakthroughs to steer and control AI systems much smarter than us. Our goal is to solve the core technical challenges of superintelligence alignment by 2027.
[1] https://jan.leike.name/ [2] https://openai.com/superalignment/
A larger proportion of people diagnosed with mental disorders report cigarette smoking compared with people without mental disorders. Among US adults in 2019, the percentage who reported past-month cigarette smoking was 1.8 times higher for those with any past-year mental illness than those without (28.2% vs. 15.8%). Smoking rates are particularly high among people with serious mental illness (those who demonstrate greater functional impairment). While estimates vary, as many as 70-85% of people with schizophrenia and as many as 50-70% of people with bipolar disorder smoke.
I mean it hides nuance in conversation.
It is not just me and it is not just the smoking: https://www.cambridge.org/core/blog/2020/08/19/physically-he...
We have known for many years that people who suffer from schizophrenia die younger than expected, as much as 20 years younger than the general population. This appears unfair, and it was the inspiration of this work. Most people thought that this added risk of death was mostly due to the higher prevalence in schizophrenia of smoking, obesity and to other lifestyle differences.
For this reason, we recruited 40 patients with schizophrenia and an equal number of healthy controls, and scanned their hearts using a state-of-the-art approach, called cardiac magnetic resonance. This was performed at the state-of-the-art Robert Steiner MRI unit.
[...] Surprisingly, in our study we found that even after matching patients and healthy controls for age, sex, ethnicity and body mass index (BMI, deriving from height and weight); and after excluding any participants with any medical conditions, and other risk factors for heart disease, people with schizophrenia show hearts that are smaller and chunkier than controls. These changes are similar to those found in aging.
You mean like PALM-E? https://palm-e.github.io/
Embodiment is the easy part.
it's a bit more complicated than that
https://www.youtube.com/watch?v=Z37i8-FnAh8
and on top of this the method of therapy is to find better copings, not just to vent.
- Patient choosing a specific mood problem/feeling they want to work on.
- A mood survey, where the patient rates their own level of e.g. anxiety, depression, fear, hopelessness. (e.g. out of 5 or 10).
- Therapy session, following his TEAMS CBT structure. Including patient choosing how much fear they'd like to feel (e.g. they want to keep a little bit of fear so they don't endanger themselves, but don't want to be overwhelmed by fear, 5% or 20%, say).
- A repeat of the mood survey, where the patient re-assesses themselves to see if anything has improved. There's no units on the measures because it's self-reported, the patient knows if the fear is unchanged, a little less, a lot less, almost gone, completely gone, and that's what matters.
That gives them feedback; if there is improvement within a session they know something in the session helped, if several sessions go by with no improvement they know it and can change things up and move away from those unhelpful approaches in future with other patients, and if there is good improvement - patient is self-reporting that they are no longer hopeless about their relationship status, or afraid of social situations, or depressed, to the level they want, then therapy can stop.
He's adamant that a single 2hr session is enough to make a significant change in many common mood disorders[2], and this "therapy needs to take 10 years" is a bad pattern and therapists who don't take mood surveys and before and after every session are flying blind. With feedback on every session and decades of experience, he has identified a lot of techniques and ways to use them which actually do help people's moods change. I liken it to the invention of test cases and debuggers (and looking at the output from them).
[1] Quick list: https://feelinggood.com/list-of-feeling-good-podcasts/ more detailed database: https://feelinggood.com/podcast-database/
[2] no, internet cynic, obviously not everything and presumably not whatever it is you have.
I encourage everyone who is skeptical of superalignment (there's a reason very smart people like Ilya were heading that team) watch Emmett Shear (former OpenAI and Twitch CEOs) talk about this problem in the discourse.
> He was writing about a tax dispute between the Pennsylvania General Assembly and the family of the Penns, the proprietary family of the Pennsylvania colony who ruled it from afar. And the legislature was trying to tax the Penn family lands to pay for frontier defense during the French and Indian War. And the Penn family kept instructing the governor to veto. Franklin felt that this was a great affront to the ability of the legislature to govern. And so he actually meant purchase a little temporary safety very literally. The Penn family was trying to give a lump sum of money in exchange for the General Assembly's acknowledging that it did not have the authority to tax it.
> It is a quotation that defends the authority of a legislature to govern in the interests of collective security. It means, in context, not quite the opposite of what it's almost always quoted as saying but much closer to the opposite than to the thing that people think it means.
https://www.npr.org/2015/03/02/390245038/ben-franklins-famou...
"The internet is for porn" https://youtu.be/LTJvdGcb7Fs?si=8H1OzeyG5XzU-Qe8
> To me, the local minima looked "good"
AI's entire business [0] is generating high quality digital content for free, but we've never ever ever needed help "generating content". For millennia we've sung songs and told stories, and we were happy with the media the entire time. If we'd never invented Tivo we'd be completely happy with linear TV. If we'd never invented TV we'd be completely happy with the radio. If we'd never invented the the CD we'd be completely happy with tapes. At every local minima of media, humanity has been super satisfied. Even if it were a problem, it's nowhere near the top of the list. We don't need more AI-generated news articles, music, movies, photos, illustrations, websites, instant summaries of research papers, (very very bad) singing. No one's looking around saying, "God there's just not enough pictures of fake waves crashing against a fake cliff". We need help with stuff like diseases and climate change. We need to figure out fusion, and it would be pretty cool if we could build the replicator (I am absolutely serious about the replicator). I remember a quote from long ago, someone saying something like, "it's lamentable that the greatest minds of my generation are focused 100% on getting more eyeballs on more ads". Well, here we are again (still?).
So why do we get wave after wave of companies doing this? Advances in this area are insanely popular and create instant dissatisfaction with the status quo. Suddenly radio is what your parents listened to, fast-forwarding a cassette is super tedious, not having instant access to every episode of every show feels deeply limiting, etc. There's tremendous profits to be had here.
You might be thinking, "here we go again, another 'capitalism just exploits humanity's bugs' rant", which of course I always have at the ready, but I want to make a different point here. For a while now the rich world has been _OK_. We reached an equilibrium where our agonies are almost purely aesthetic: "what kind of company do I want to work for", "what's the best air quality monitor", "should I buy a Framework on a lark and support a company doing something I believe in or do the obvious thing and buy an MBP", "how can I justify buying the biggest lawnmower possible", etc. Barring some big dips we've been here since the 80s, and now our culture just gasps from one "this changes everything" cigarette to the next. Is it Atari? Is it Capcom? Is it IMAX? Is it the Unreal Engine? Is it Instagram? Is it AI? Is it the Internet? Is it smartphones? Is it Web 2.0? Is it self-driving cars? Is it crypto? Is it the Metaverse and AR/VR headsets? I think us in the know wince whenever people make the leap from crypto to AI and say it's just the latest Silicon Valley scam--it's definitely not the same. But the truth in that comparison is that it is just the next fix, we the dealers and American culture the junkies in a codependent catastrophe of trillions wasted when like, HTML4 was absolutely fine. Flip phones, email, 1080p, all totally fine.
There is peace in realizing you have enough [1]. There is beauty and discovery in doing things that, sure, AI could do, but you can also do. There is joy in other humans. People listening to Hall & Oates on Walkmans teaching kids Spanish were just as happy (actually, probably a lot happier) as you are, and assuredly happier than you will be in a Wall-E future where 90% of your interactions are with an AI because no human wants to interact with any other human, and we've all decided we're too good to make food for each other or teach each other's kids algebra. It is miserable, the absolute definition of misery: in a mad craze to maximize our joy we have imprisoned ourselves in a joyless, desolate digital wasteland full of everything we can imagine, and nothing we actually want.
[0]: I'm sure there's infinite use cases people can come up with where AI isn't just generating a six fingered girlfriend that tricks you into loving her and occasionally tells you how great you would look in adidas Sambas. These are all more cases where tech wants humanity to adapt to the thing it built (cf. self-driving cars) rather than build a thing useful to humanity now. A good example is language learning: we don't have enough language tutors, so we'll close the gap with AI. Except teaching is a beautiful, unique, enriching experience, and the only reason we don't have enough teachers is that we treat them like dirt. It would have been better to spend the billions we spent on AI training more teachers and paying them more money. Etc. etc. etc.
[1]: https://www.themarginalian.org/2014/01/16/kurt-vonnegut-joe-...
I don't think so. From the original text [1]:
"In fine, we have the most sensible Concern for the poor distressed Inhabitants of the Frontiers. We have taken every Step in our Power, consistent with the just Rights of the Freemen of Pennsylvania, for their Relief, and we have Reason to believe, that in the Midst of their Distresses they themselves do not wish us to go farther. Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety.
"Our Assemblies have of late had so many Supply Bills, and of such different Kinds, rejected on various Pretences; Some for not complying with obsolete occasional Instructions (tho’ other Acts exactly of the same Tenor had been past since those Instructions, and received the Royal Assent;) Some for being inconsistent with the supposed Spirit of an Act of Parliament, when the Act itself did not any way affect us, being made expresly for other Colonies; Some for being, as the Governor was pleased to say, “of an extraordinary Nature,” without informing us wherein that extraordinary Nature consisted; and others for disagreeing with new discovered Meanings, and forced Constructions of a Clause in the Proprietary Commission; that we are now really at a Loss to divine what Bill can possibly pass."
"we see little Use of Assemblies in this Particular; and think we might as well leave it to the Governor or Proprietaries to make for us what Supply Laws they please, and save ourselves and the Country the Expence and Trouble."
"And we being as desirous as the Governor to avoid any Dispute on that Head, have so framed the Bill as to submit it entirely to his Majesty’s Royal Determination, whether that Estate has or has not a Right to such Exemption."
Also, while that's the first appearance of the quote, it's not the last time he used it. He also reiterated it as an envoy to England during negotiations to prevent the war [2].
Additionally, a similar quote was from well before either in Poor Richard's Almanac in 1738, that also illustrates his thinking [3] and shows that he was well aware of the plain meaning of what he was saying, it certainly wasn't limited to a tax dispute:
"Sell not virtue to purchase wealth, nor Liberty to purchase power."
[1] https://founders.archives.gov/documents/Franklin/01-06-02-01...
[2] https://oll.libertyfund.org/quotes/benjamin-franklin-on-the-...
My intuition leads me to believe that these are arising properties/characteristics of complex and large prediction engines. A sufficiently good prediction/optimization engine can act in an agentic way, while never had that explicit goal.
I recently read this very interesting piece that dives into this: https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...
https://www.nytimes.com/2023/02/16/technology/bing-chatbot-m...
https://www.youtube.com/watch?v=VihA_-8kBNg&t=17m35s
So I went and checked the letter. Here is the letter with signatories, I don't see his name in the 'J' section (alphabetized by first name):
I don't see a longer name for him that doesn't start with Jan anywhere else I could find. I think that tweet was the same day as the letter though, so maybe he's on there and I just didn't see it, he meant something more complicated and didnt agree with all of the letter, or he missed the deadline for signing the letter.
https://x.com/janleike/status/1791498174659715494
Got to wonder what the actual trigger point for his & Ilya's decision was? For the two of them to leave at the same time suggests an event more than just incremental unhappiness with lack of support for safely/alignment. Perhaps a release decision for GPT-4's successor has been made that they disagree with?