zlacker

Yes, that's part of the reason why alignment is such a huge problem.

You can imagine an AI that answers questions and helps you get things within reason that doesn't hurt anyone else plus corrections for whatever problems you imagine with this. That's roughly an aligned AI. It will help you build a bomb as a fun experiment, but would stop you from hurting someone with the bomb.

>>ALittl+(OP)
Apart from some obvious cases that everyone agrees with, alignment is not a big problem it is an incoherent one. It can’t be “solved” any more than the problem of what the best ice cream flavor is can be solved.

Humanity doesn’t have unified interests or shared values on many things. We have different cultural memories and different boundaries. What to some is an expression of a fundamental right is an affront.

>>janals+5b
At the limit sure there’s variance, but our shared selected history has a lot in common, something a non-human intelligence would not get for free: https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden...

I’m also not a moral relativist, I don’t think all values are equivalent, but you don’t even need to go there - before that point a lot of what humans want is not controversial and the “obvious” cases are not so obvious or easy to classify.