zlacker

>>tim_sw+(OP)
Why are they starting to sound more and more cult-like? This is an incredibly unscientific blog post. I get that they are a private company now, but why even release something like this without further details?

>>skepti+pe
Because the "AGI" pursuit is at least as much a faith movement as it is a rational engineering program. If you examine it more deeply, the faith object isn't even the conjectured inevitable AGI, it's exponential growth curves. (That is of course true for startup culture more generally, from which the current AI boom is an outgrowth.) For my money, The Singularity is Near still counts as the ur-text that the true believers will never let go, even though Kurzweil was summarizing earlier belief trends.

It's just a pity that the creepy doomer weirdos so thoroughly squatted the term "rationalist." It would be interesting to see the perspective on these people 100 years hence, or even 50. I don't doubt there will still be remnant believers who end up moderating and sanitizing their beliefs, much like the Seventh Day Adventists or the Mormons.

>>batman+vn
you don't need to believe in exponential growth per se, all you really need to believe is that humans aren't that capable relative to what could be in principle built - it's entirely possible logistic growth may be more than enough to get us very far past human ability once the right paradigm is discovered.

>>zzzzzz+7F
Exactly. All that is needed for AGI to eventually be developed, is that humans do NOT have some magical or divine essence that set us apart from the material world.

Now the _timeline_ of AGI could be anything from the a few years to millennia, at least if evaluated 40 years ago. Now, though, it really doesnt seem very distant.

>>trasht+Me1
That's going to be the key principle of the new religion invented by AI: that humans are just like machines, and AI is the supreme machine.

>>akomtu+1p2
Well, assuming superintelligence emerges AND we don't find any evidence of anything supernatural inside human brains, what does "machine" even mean at that point?

Anyway, when/if AIs start to create the narrative to the extent they can author our religion, they already have full control. At least if some AI decides to become a God with us as worshippers, at least we stay around a bit longer.

Probably a better outcome than if it decides that it has better uses for the atoms in our bodies.

In fact, this may even be a solution to the alignment problem. An all-powerful but relatively harmless (non-mutating, non-evolving, non-reproducing) single AI that creates and enforces a religion that prevents us from creating dangerous AI's, weapons of mass destruction, gray goo or destroy the planet, while promoting pro-social behaviour while otherwise leaving us free to do mostly what we want to do.

>>trasht+Tr2
A distinct property of a machine is determinism. Machines are made of electrons that obey the wave equation: it states that all electrons evolve as one. However we skillfully limit this waveness and squash electrons into super deterministic 01 transistors. We need it because our computing paradigm revolves around determinism. A more general theory that works with probabilities and electrons as they are hasn't been developed yet. And here lies the danger: if a super deterministic AI becomes the ruler of our society, it will prevent further growth. The same phenomenon on individual level is called arrested development: when the individual over-develops a minor skill, becomes obsessed with it and it blinds him from exploring anything else.

A still possible future is when science makes a breakthru in quantum computing and finds a way for humans to steer AI with their minds: it would be the neuralink in reverse. This would force science to research the true nature of that connection which will help to avert the AI doom.