value learning
1 article
Value learning is the effort to have an AI system work out what people want from evidence, instead of being handed a fixed goal written by a programmer.
The motivation is that human values are hard to write down. A goal stated in code or in a sentence always leaves things out, and a capable system will exploit the gaps, reaching the stated target in ways nobody wanted. Learning the goal from people is meant to close that gap.
Several routes exist. A system can watch what people do and infer the aims behind the behavior, an approach called inverse reinforcement learning. It can ask people to compare two outcomes and learn from which one they prefer. It can learn from the enormous record of human writing, which carries a great deal of information about what people care about. One influential proposal treats the machine as uncertain about the human's goal, so that it has a reason to ask, to defer and to accept correction. Each route has weak points. People act against their own interests, so behavior is an imperfect guide. Stated preferences change with how the question is framed. People disagree with each other, and nobody is sure whose values should count or how to combine them.
The problem matters because the more capable a system is, the more its goals decide what happens. A slightly wrong objective in a weak system is a bug. In a very powerful one it could shape the world in ways that are hard to undo.