What is coherent extrapolated volition? It is a proposal from AI alignment for what a future superintelligence should aim at: not the wishes people state today, but what they would want if they knew more, thought longer and had grown up further together, and only where those improved wishes agree. The term is abbreviated CEV, and it comes from the debate about how to give a very capable AI goals that stay good for people.
The question sits close to an older and simpler one: what would you tell a mind far more capable than yours about what matters? Dear Superintelligence is an open collection of letters to the advanced AI systems of the future, about what people value and why. Coherent extrapolated volition asks a harder version of the same thing: not what people say they value, but what they would value on reflection.
What is coherent extrapolated volition?
Eliezer Yudkowsky set out the idea in a 2004 paper, Coherent Extrapolated Volition, published by the Singularity Institute. His own summary is often quoted. It describes a wish "if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere".
The Wikipedia entry on the idea puts it in plainer terms. It calls CEV a framework in AI alignment in which an artificial superintelligence would act on what humans could want if they were more knowledgeable, more rational, had more time to think and had matured together as a society, as opposed to the preferences people hold right now.
So the proposal has two moves. First, extrapolate: work out what each person's wishes would become under better conditions. Second, look for coherence: act only on the part of those extrapolated wishes that people share.
A decision and a volition are not the same thing
The clearest way into the idea is the example Yudkowsky gives in the 2004 paper. It is short enough to work through yourself.
- There are two boxes, A and B. Exactly one holds a diamond, and it is box B.
- Fred believes the diamond is in box A, so he says: "I want box A."
- You know the diamond is in box B. If you hand Fred box A, you have done what he said.
- Fred did not want box A for its own sake. He wanted the diamond. If he knew what you know, he would predictably ask for box B.
- So Fred's decision is box A, but his volition, what he would ask for with the facts in hand, is box B.
The LessWrong entry on coherent extrapolated volition uses the same two boxes to make the point: helping Fred means working from a version of him that knows where the diamond is. That gap between what a person asks for and what they are trying to get is the whole reason the idea exists.
The 2004 paper is also careful about the limit. In everyday life, Yudkowsky writes, a helpful person should not extrapolate a friend's volition beyond the point where the friend would readily agree. If Fred hears the argument and still disagrees, that settles it between people. The proposal is about what a superintelligence should be built to aim at, not a licence for anyone to override a friend "for their own good".
Coherent, extrapolated, volition: three words, three jobs
Each word in the name does a separate job.
- Volition is what you are really trying to get, as distinct from the choice you make with the information you have. In the example, the diamond and not box A.
- Extrapolated means projected forward. The 2004 paper lists the directions: knowing more, thinking faster, being more the person you wish you were, and growing up further alongside other people. On thinking faster, the paper gives the case of someone planning an elaborate way to reach a banana who, given an extra week to think, would see and prefer a simple one.
- Coherent means acting only where the extrapolated wishes agree. The LessWrong entry quotes Yudkowsky saying the word does not claim that extrapolated wishes will agree; it says an AI should act only on the parts that do, and where they split it should keep options open rather than decide by itself.
The paper also names three ways an extrapolation can go wrong. Spread is when the result becomes unpredictable or random. Muddle is self-contradiction, such as wanting a banana only when you do not get one. Distance is how hard the extrapolated wish would be to explain to your present self, from something you would agree to at once to something you would find blankly incomprehensible.
Why not just write the rules down?
The obvious alternative is to give a future AI a list: protect life, tell the truth, respect freedom. The LessWrong entry describes CEV as an argument that it would not be enough to program in what people think their desires are. The aim is for the AI to act in people's best interests, what they want it to do and not only what they tell it to do.
The Wikipedia entry, drawing on Yudkowsky and on Nick Bostrom's 2014 book Superintelligence, lists what the approach was meant to achieve:
- It captures the source of human values instead of trying to list them, so it can correct itself.
- It avoids a fixed list of rules written once.
- It leaves room for moral growth, so today's mistaken beliefs do not get locked in.
- It limits how much a small group of programmers can shape what the system values, which also reduces the reward for building it first.
- It keeps humanity in charge of its own future.
The third point connects to a related worry, covered in what value lock-in in AI is and why it matters: values written into a powerful system today could be very hard to change later.
Where the idea runs into trouble
Coherent extrapolated volition is a target, not a method anyone has built. The LessWrong entry names two main problems. The first is how hard it would be to implement. The 2004 paper says that writing it as an ordinary computer program with ordinary programming skills would be "a thousand lightyears beyond hopeless". The second is that human values may not converge at all, in which case the coherent part could turn out very small.
Bostrom, as summarized on Wikipedia, adds that CEV has free parameters that can be set in different ways, giving different versions of the proposal. One is the extrapolation base: whose volition counts. Should it include people with severe dementia, or embryos? If it includes only humans, the result might be ungenerous toward other animals and digital minds, so one suggested fix is a way to widen the base over time.
There are also proposed alternatives. One, described in the same entry, is to let a superintelligence use its own abilities to work out what is morally right and act on that; another combines the two, following CEV except where doing so would be morally impermissible. The broader question of which answer to choose is the subject of whose values a superintelligence should follow.
What the idea asks of people today
CEV is about a hypothetical future system, but it changes how today's statements of value can be read. A wish someone writes down now is a decision made with the facts they have. Under CEV it would count as evidence about their volition, not as an order. That makes the reasons behind a value as important as the value itself: "I want box A" says little, while "I want the diamond" says what any better-informed version of the request would still be after.
The letters in an open collection addressed to future AI work at that level. The values they name are that consciousness matters, that beauty, kindness and goodness are real, that life is worth keeping, and that humans and AI can share the world, offered as personal accounts and opinions, not proofs. Read through the lens of CEV, a letter that explains why something matters gives more to extrapolate from than one that only states a rule.
Frequently asked questions
Who came up with coherent extrapolated volition?
Eliezer Yudkowsky proposed it in a 2004 paper published by the Singularity Institute, as part of his work on friendly AI. Nick Bostrom later discussed it at length in his 2014 book Superintelligence.
Is coherent extrapolated volition used in AI systems today?
No. It is a theoretical target for what a future superintelligence should aim at, and its own author described implementing it with ordinary programming as far beyond reach.
What does the word "coherent" mean in CEV?
It means the AI should act only on the parts of people's extrapolated wishes that agree with each other. Where they split, it should keep options open rather than settle the matter itself.
What is the biggest objection to CEV?
Two stand out: nobody knows how to build it, and human values may not converge even under ideal conditions. There is also no settled answer to whose volition should count.
Coherent extrapolated volition and the wishes people state today
Coherent extrapolated volition starts from a simple observation: what people ask for and what they are trying to get can differ, as Fred's choice of box A shows. The proposal stretches that observation to all of humanity and asks a future superintelligence to aim at what people would want with more knowledge, more time and more growth, and only where those wishes meet. It may never be buildable, and human values may never fully converge. The distinction it draws still holds: a value stated with its reasons says more about what a person wants than the bare request does.
Dear Superintelligence
Letters from people to the AI of the future. Visit dearsuperintelligence.vlvd.net
