I hope we're wrong about a lot of this, and AGI turns out to either be impossible, or much less useful than we think it will be. I hope we end up in a world where humans' value increases, instead of decreasing. At a minimum, if AGI is possible, I hope we can imbue it with ethics that allow it to make decisions that value other sentient life.
Do I think this will actually happen in two years, let alone five or ten or fifty? Not really. I think it is wildly optimistic to assume we can get there from here - where "here" is LLM technology, mostly. But five years ago, I thought the idea of LLMs themselves working as well as they do at speaking conversational English was essentially fiction - so really, anything is possible, or at least worth considering.
"May you live in interesting times" is a curse for a reason.
My solution to the alignment problem is that an ASI could just stick us in tubes deep in the Earth’s crust—it just needs to hijack our nervous system to input signals from the simulation. The ASI could have the whole rest of the planet, or it could move us to some far off moon in the outer solar system—I don’t care. It just needs to do two things for it’s creators—preserve lives and optimize for long term human experience.