zlacker

When we talk about developing models in alignment with goals to herd or control super intelligence since humans will not be able to do it, we are necessarily talking about designing something capable of governing and controlling very complex systems. What happens when a "misaligned system" happens to be a rebellious human being? In trying to find ways to control potential superintelligence and keep it on a positive path, is it possible we are building the very system that enslaves us?

Maybe this is one of those problems.

>>friend+(OP)
Good point. Look at it this way, though: If such a system ends up stopping a person that is about to eradicate humanity using nuclear weapons, bioweapons or similar, is that a good or bad thing?

>>trasht+r1
I don't know. On the one hand there's the idea that humanity should be left to take it's organic path. The machine can't know that guy's grievances. Do we really want to be kept as pets? Or is it just another tool being used to protect people? I guess it really depends on the role it plays in the social structure directly.

>>friend+l72
> Do we really want to be kept as pets?

You could see us as pets at this points, or you can see us as the aging grandparents who need help with anything invented after 1980.