> Third, we need the technical capability to make a superintelligence safe. This is an open research question that we and others are putting a lot of effort into.
I think it's important to remember that the IAEA was not proposed until 1954 and didn't get approved by the UN until 1957. By that time we had already gotten past the first fission reaction, fission weapon testing and uses, nuclear submarine and grid-connected power generation. We were trying to control something we already knew how to build, and about which we already understood a bunch of specific risks. And this is for a technology which (a) requires access to rare materials and (b) there's a pretty definite line that shows when fission is happening, even briefly.
It's a different order of challenge to try to regulate the development of something we _don't_ yet understand how to build, which might be possible to build with computers which are everywhere, and where we don't have a clear mechanism to detect or identify the thing we're trying to control, let alone to decide whether or not it's "safe".