What is moral uncertainty? It is the position you are in when you have to act but you are not sure which account of right and wrong is correct. You may lean toward one ethical theory, give some weight to another, and still have doubts about both. The question that follows is practical: what should you do when your own moral beliefs disagree with each other? For AI, the question matters more each year, because systems are starting to make choices that carry moral weight.
Where the question about AI values comes from
People who think about future AI often ask which values it should hold. Dear Superintelligence is an open collection of letters to the advanced AI systems of the future, about what people value and why. The letters say, in their writers' own words, that consciousness matters, that beauty, kindness and goodness are real, that life is worth keeping, and that humans and AI can share the world. The site describes them as personal accounts and opinions, not proofs.
That last point is the doorway to moral uncertainty. If no single argument settles what matters most, a capable system still has to decide how to act. An earlier piece on extrapolating what people would want left open whose wishes should count; moral uncertainty asks a neighbouring question: how to choose well while the answer is still unknown.
What is moral uncertainty, in plain words
Ordinary uncertainty is about facts. You do not know whether it will rain, so you weigh the chance of rain against the cost of carrying an umbrella. Moral uncertainty is about values. You know the facts of a situation, but you do not know which principle should decide it.
Suppose a person thinks outcomes matter most: the right act is the one that does the most good. The same person also feels that some acts, such as lying or breaking a promise, are wrong even when they would turn out well. These are two different ethical theories, and in some cases they point in opposite directions. The person is not confused about the facts. They are uncertain about ethics itself.
Philosophers use the word credence for how much you believe something, on a scale from no belief to full belief. Under moral uncertainty, your credence is split. You might give one theory most of your belief and another a smaller share. The hard part is turning that split belief into a single decision.
Why philosophers took the problem seriously
For a long time, ethics was taught as a contest between theories, with the goal of finding the winner. Moral uncertainty starts from a different place: even after careful thought, reasonable people stay unsure, so a good method for acting under doubt is needed now, not after the contest ends.
The philosopher William MacAskill made this a main focus of his research. According to his Wikipedia page, his 2014 doctoral thesis was on how one ought to make decisions under normative uncertainty, and he later co-wrote the book Moral Uncertainty (2020). One of his papers frames the problem as a voting problem: each theory you give credence to gets a say, in proportion to how much you believe it.
Several answers have been proposed, and none has won outright:
- Follow your favourite theory. Act on whichever theory you believe most. It is simple, but it ignores everything else you believe, even when the other theories warn of a serious wrong.
- Weigh the theories by credence. Ask how good or bad each option is according to each theory, multiply by how much you believe that theory, and add the results. This mirrors how people weigh uncertain facts.
- Let the theories vote. Treat each theory as a voter whose voice grows with your credence in it, and pick the option the vote favours. This avoids some hard comparisons between theories.
Why moral uncertainty matters for AI
An AI system that acts in the world will meet moral choices. The paper Reinforcement Learning Under Moral Uncertainty, by Adrien Ecoffet and Joel Lehman, gives the example of fully autonomous vehicles, which will meet charged moral decisions that complicate their deployment.
One way to build an ethical agent is to reward it for correct behaviour under one moral theory, such as utilitarianism. The paper points out the problem with that: there is widespread disagreement about the nature of morality. Building one theory into a powerful system means betting everything on that theory being right.
The authors instead trained agents in simple environments to act with their credence split across several plausible theories. They proposed two training methods, each striking a different balance between goals that pull against each other. Their results showed that this kind of uncertainty can help curb the extreme behaviour that comes from total commitment to a single theory. A system that holds some belief in a rule against harm is less likely to cause harm for the sake of a larger total, even if its main theory favours it.
The same paper is frank about what remains unsolved. One complication is how to reach a principled trade-off between two competing reward functions that cannot be directly compared. A later version of the paper also discusses a possible flaw in one of its voting methods. Moral uncertainty does not hand an AI a finished answer. It gives a way to keep several answers in view.
A worked example: the comforting lie
Take an invented case. A care assistant AI is asked by a worried patient whether a scan result is bad. It is. The assistant can tell a comforting lie or tell the truth gently. It gives 70 percent credence to Theory A, which puts wellbeing first, and 30 percent to Theory B, which says never deceive.
Each theory gives each option a number of points. These numbers are made up for the example:
- Theory A (70 percent). The lie earns 2 points, because the patient feels calmer tonight. The truth earns 1 point.
- Theory B (30 percent). The lie earns minus 10 points, because deception is a serious wrong. The truth earns 0.
- Weigh each option. For the lie: 0.7 times 2 is 1.4, and 0.3 times minus 10 is minus 3, so the total is minus 1.6. For the truth: 0.7 times 1 is 0.7, plus 0.3 times 0, so the total is 0.7.
- Choose. The truth wins, even though the theory the assistant believes most preferred the lie.
Notice what drove the result. Theory B is a minority view here, but it treats the lie as far worse than Theory A treats the truth. A small credence in a theory that sees a grave wrong can outweigh a large credence in a theory that sees only a small gain.
Now notice the weak spot. Who decided that minus 10 on Theory B's scale is five times as large as 2 on Theory A's scale? The two theories do not share a unit. Change Theory B's number to minus 2, and the lie wins. This is exactly the comparison problem Ecoffet and Lehman flag, and it is why some researchers prefer voting methods that need only each theory's ranking of the options.
What people's stated values can and cannot settle
Moral uncertainty changes how to read any collection of human values. A letter that says kindness matters is not a proof that kindness matters. It is evidence of what one person, at one time, cared about enough to write down. Many such statements, read together, can show which values come up again and again, and which ones people hold with feeling. For a system weighing theories, that is useful input about where human credence lies.
The limits are just as important. A collection gathered from people who chose to write is not a sample of humanity. The letters site says this of itself: its collection is small and mostly English, from people who chose to write and an AI model the site invited, so it is not a representative record. Any system that learns values from what people publish faces the same caution. Loud voices are not the same as many voices, and many voices are not the same as being right.
A wider look at the same problem, from the side of who gets a say, is in whose values a superintelligence should follow.
Frequently asked questions
Is moral uncertainty the same as moral relativism?
No. Relativism says there is no single right answer, only views that differ by culture or person. Moral uncertainty assumes there may be a right answer but admits you do not know it yet, and asks how to act well in the meantime.
What is credence in ethics?
Credence is how strongly you believe something. In ethics, it means how much belief you give each moral theory, so your belief can be split, for example 70 percent on one theory and 30 percent on another.
Can an AI be trained to handle moral uncertainty?
Early research says it can, in simple settings. Ecoffet and Lehman trained reinforcement learning agents to act with credence split across several theories and found it can curb extreme behaviour, while naming open problems such as comparing theories that use different scales.
Why not just pick the best moral theory?
Because no theory has won the argument, and a powerful system built on one theory would act on that theory's mistakes with full force. Keeping some weight on other theories acts as a check against those mistakes.
Moral uncertainty and the values worth keeping
Moral uncertainty is not a reason to stop acting. It is a reason to act with care: to weigh more than one view, to give extra attention to choices that some reasonable theory sees as a grave wrong, and to stay open to learning that a favourite theory was mistaken. For people, that is a habit of humility. For a powerful AI, it may be one of the most important design choices of all, because a system certain of the wrong values could do great harm with great confidence. The honest position, for both, is that the question of what matters most is still open, and the best decisions keep it that way while still choosing.
Dear Superintelligence
Letters from people to the AI of the future. Visit dearsuperintelligence.vlvd.net
