Setting the world on fire and disrupting societies gleefully, while basically building bunkers (figuratively more than literally) and consolidating surveillance and propaganda to ride out the cataclysm, that's what I'm seeing.
And the stories to sell people on continuing to put up with that are not even good IMO. Just because the people who use the story to consolidate wealth and control are excited about that, we're somehow expected to be excited about the promise of a pair of socks made from barbed wire they gave us for Christmas. It's the narcissistic experience: "this is shit. this benefits you, not me. this hurts me."
One thing is sure, actual intelligence, regardless of how you may define it, something that is able to reason and speak freely, is NOT what people who fire engineers for correcting them want. It's not about a sort of oracle for humanity to enjoy and benefit from, that just speaks "truth".
I don’t believe the control problem is solved, but I’m not sure it would matter if it is.
I don't even understand what the proposed mechanism for "rouge AI enslaves humanity" is. It's scifi (and not hard scifi) as far as I can see.
To put it simply, it could outcompete humanity on every metric that matters, especially given recent advancements in robotics.
Again, I wonder why no group of smart people with brilliant ideas has unilaterally imposed those ideas on the rest of humanity through sheer force of genius.
Having said that, we do not to understand the world to exploit it for ourselves. And what better way to understand and exploit the universe than science? Its an endearment.
I don't know if this will happen with any certainty, but the general idea of commoditising intelligence very much has the ability to tip the world order: every problem that can be tackled by throwing brainpower at it will be, and those advances will compound.
Also, the question you're posing did happen: it was called the Manhattan Project.
We already did. Look at the state of animals today vs <1 mya. Bovines grown in unprecedented mass numbers to live short lives before slaughter. Wolves bred into an all new animal, friendly and helpful to the dominate species. Previously apex predators with claws, teeth, speed and strength, rendered extinct.
“AGI” has proven to be today’s hot marketing stunt for when you need to raise another round of cash and your only viable product is optimism.
Flying cars were just around the corner in the 60s, too.
I think reinforcement learning with little to no human feedback, O-1 / R-1 style, might be that revolution.
All of these entities would have been enormously more powerful with access to an AGI's immortality, sleeplessness, and ability to clone itself.
If Alexander could have left perfectly aligned copies of himself in every city he passed, he could have gotten much more control and authority, and still avoided a fight by agreeing to maintain the local power structure with himself as the new head of state.
The problem you're proposing could be solved via a high quality cellular network.
1)Leibniz wasn't superhuman 2) Leibniz couldn't work 24/7 3) he could not self increase the speed of his own hardware (body) 4) he could not spawn 1 trillion copies of him to work 24/7
Like how much time did you think before writing this
A trillion hyperintelligent demons might be cogitating right now on the head of a pin. You can't prove they aren't thinking up all sorts of genius evil schemes. My point is that "intelligence" has never been a sufficient - or even necessary - component of imposing ones will on humans.
I feel like HN/EA/"Grey Tribe" people fail to see this because they so worship intellect. I'm much more likely to fall victim to a big dumb man than smart computers.
The Manhattan Project would be a cute example if the Los Alamos scientists had gone rogue and declared themselves emperors of mankind, but no, in fact the people in charge remained the people in charge - mostly not supergeniuses.