zlacker

What does "properly aligned" even mean? Democracies even with countries don't have alignment, let alone democracies across the world. They're a complete mess of many conflicting and contradictory stances and opinions.

This sounds, to me, like the company leadership want the ability to do some sort of picking of winners and losers, bypassing the electorate.

>>mlindn+(OP)
Any AGI must at a minimum be aligned with these two values:

(1) humanity should not be subjugated

(2) humanity should not go extinct before it’s our time

Even Kim Jong Un would agree with these principles.

Currently, any AGI or ASI built based on any of the known architectures contemplated in the literature which have been invented thus far would not meet a beyond-a-reasonable-doubt standard of being aligned with these two values.

>>mlindn+(OP)
> What does "properly aligned" even mean?

You know those stories where someone makes a pact with the devil/djin/other wish granting entity, and the entity does one interpretation of what was wished, but since it is not what the wisher intended it all goes terribly wrong? The idea of alignment is to make the djin which not only can grant wishes, but it does them according to the unstated intention of the wisher.

You might have heard the story of the paper clip maximiser. The leadership of the paperclip factory buys one of those fancy new AI agents and asks it to maximise paperclip production.

What a not-well aligned AI might do: Reach out through the internet to a drug cartel’s communication nodes. Hack the communications and take over the operation. Optimise the drug traficking operations to gain more profit. Divert the funds to manufacture weapons for multiple competing factions on multiple crisis points on Earth. Use the factions against each other. Divert the funds and the weapons to protect a rapidly expanding paperclip factory. Manipulate and blackmail world leaders into inaction. If the original leaders of the paperclip factory try to stop the AI eliminate them, since that is the way to maximise paper clip production. And this is just the begining.

What a well alligned AI would do: Fine tune the paperclip manufacturing machinery to eliminate rejects. Reorganise the factory layout to optimise logistics. Run a succesfull advertising campaign which leads to a 130% increase in sales. (Because clearly this is what the factory owner intended it to do. Altough they did a poor job of expressing their wishes.)

>>upward+x9
I think this is a crazy set of values.

'.. before it's our time' is definitely in the eye of the beholder.

>>krisof+Cc
I like your extremist example, however I fear what "properly aligned" means for more vague situations, where it is not at all clear what the "correct" path is, or worse, that it's very clear what "correct" is for some people, but that "correct" is another man's "evil".