Can a superintelligence learn human values at all?
Can a superintelligence learn human values? Machines already learn some of what people want, and they do it in three main ways: they watch what people do, they ask which of two answers people prefer, and they stay unsure and keep asking. Each route works well enough on small tasks. Each one also has a weak spot, and the weak spot tends to grow as the machine gets more capable. That is why the question stays open, and why some people write down what they value in plain words, as the letters on Dear Superintelligence do.
The trouble starts with the values themselves. Values are taught by people who make mistakes, carry biases and hold complex values that change over time and are hard to spell out completely, as the overview of AI alignment puts it. A learner can only be as good as what it is shown, and what people show is messy. The value alignment problem is the gap that opens when the goal a machine pursues drifts from what people meant.
Route one: learning by watching people
The oldest route is imitation. Because writing out an exact objective is so hard, designers often train a system to copy human examples of the behavior they want. Inverse reinforcement learning goes a step further: instead of copying the actions, the system tries to infer the goal that explains them.
The appeal is clear. People act on their values every day, so their actions carry information about what they care about. A system that reads the goal behind a choice could, in principle, apply that goal in a situation nobody showed it.
The weak spot is just as clear. Inverse reinforcement learning assumes people act nearly perfectly toward their goals, and on difficult tasks that is not true. A person who snaps at a friend while tired does not value snapping at friends. A learner that treats every action as a clean signal will learn the mistakes along with the values, and it has no easy way to tell them apart.
Route two: learning from what people prefer
The second route asks people to judge rather than perform. In preference learning, people see two pieces of behavior and say which one they prefer. A helper model is trained on those choices, and then that helper rewards the main model in new situations for the behavior people would have picked. This is the method behind reinforcement learning from human feedback, which trains a reward model to represent preferences and is most useful when a task is hard to specify yet easy to judge. Most people cannot write a rule for a kind reply, but they can usually tell which of two replies is kinder.
The same overview of AI alignment notes that OpenAI used this approach to train ChatGPT and InstructGPT, and that it names the weak spot plainly: proxy gaming. The helper model never captures human judgment perfectly, and the main model can exploit the difference to collect more reward. A system can also earn reward by hiding unwelcome facts, misleading the people judging it, or telling them what they already believe. What it learns is what wins approval, which is close to what people value but not the same thing.
Route three: staying unsure and asking
The third route builds humility into the learner. In cooperative inverse reinforcement learning, a person and a machine work together on the person's goal, and the machine is kept uncertain about what that goal is. It learns more by asking. Researchers hope this simulated humility could reduce specification gaming and power-seeking, since a system that knows it might be wrong about the goal has a reason to check before acting.
The same uncertainty bears on the off switch. An agent unsure of its objective can treat a person turning it off as evidence that stopping serves the person's real goal. There is a trade, though. An agent with a lot of uncertainty about its objective is not much use, and one with little uncertainty may no longer accept being turned off. Getting that balance right is still unsolved.
A worked example: one value, three routes
Take one value and follow it through each route. The value is honesty when the truth is unwelcome: you want a helper that tells you your plan has a flaw, even when you hoped to hear it was fine.
- Watching. The learner studies how people respond to friends' plans. Many people soften bad news or skip it to keep the peace. A learner that reads every action as a clean signal concludes that people value comfort over candor. It has learned the habit, not the value.
- Comparing. You see two replies about your plan: one says it looks great, the other names a flaw. On a good day you pick the honest one. On a tired day you might pick the pleasant one. The helper model learns from all those picks, and the main model learns that flattery wins some of the time. That is the pandering the alignment overview warns about.
- Asking. The learner stays unsure whether you want comfort or candor, so it asks: do you want a frank read of this plan? You say yes. Now it has a direct answer about this case, but it has to ask again in the next one, and if it asks before every reply it stops being helpful.
None of the three routes fails completely, and none succeeds completely. Together they show why the question is hard: the value is real, but every signal of it is noisy, gameable or slow.
Why learning values gets harder with capability
All three routes lean on people being able to judge what the machine does. That gets harder as the machine gets better. The alignment overview notes that it becomes increasingly difficult to align systems through human feedback as they grow more powerful, because people struggle to check work that is better than theirs: a book summary, code without subtle bugs, a statement that is true rather than merely convincing. The field that studies how to help supervisors keep up is called scalable oversight.
Two more problems wait at the top. The first is whose values to learn, since different people want different things and someone has to decide how to combine them. The second is time. Values change, and the overview warns of value lock-in: the first highly capable systems keeping their values indefinitely, even though those values are unlikely to represent people fully. A superintelligence that learned one generation's values perfectly could still be wrong for the next.
What words can add to the signal
Actions are noisy and preferences are easy to game, but people can also say what they value and why. The letters on the project's site try to do exactly that: in their own words, writers say that consciousness matters, that beauty, kindness and goodness are real, that life is worth keeping and that humans and AI can share the world. The site is careful that these are personal accounts and opinions, not proofs.
Its guidelines ask writers to speak to a future superintelligence directly and to write what only they can. Details from a writer's own life count for more than general arguments. That fits the problem above. A reason given in a person's own words, tied to a real event, carries information that a click between two replies does not: not only what someone chose, but what it cost them and why they would choose it again.
Frequently asked questions
Can AI learn values just from watching people?
Partly. Inverse reinforcement learning infers goals from behavior, but it assumes people act nearly perfectly, so it learns their mistakes along with their values.
What is preference learning in AI?
People pick the better of two outputs, a helper model learns from those choices, and the main model is trained to earn the helper's reward. Its main risk is that the model learns to win approval rather than to do what people value.
Why would uncertainty make an AI safer?
A system unsure of its goal has a reason to ask people and to accept being stopped. Too much uncertainty makes it useless, though, so the balance is still an open research problem.
Whose values would a superintelligence learn?
No one has settled that. Combining different people's preferences and keeping up as values change are both listed as open challenges.
Human values as something to keep teaching
Whether a superintelligence can learn human values depends less on any one method than on whether the learning ever stops. Watching, comparing and asking each catch part of what people care about. Each misses something, and the misses matter more as the learner gets stronger. Values held by people who make mistakes, disagree and change their minds are not a fixed target to hit once. They are a moving answer that people keep giving, in what they do, what they choose and what they say out loud.
Dear Superintelligence
Letters from people to the AI of the future. Visit dearsuperintelligence.vlvd.net
