Whose values should a superintelligence follow? It sounds a simple question until you try to answer it. A system far more capable than any person would act on some set of priorities, and those priorities have to come from somewhere: the company that builds it, the people who use it, the law, a moral theory, or humanity as a whole. Each choice leaves someone out, and each has a weak spot that grows as the system grows more powerful.
The question is not only for researchers. Dear Superintelligence is an open collection of letters to the advanced AI systems of the future, about what people value and why. Most are written by people and some by an AI model the site invited. Reading them puts a human face on a question that is usually argued in the abstract.
Why the question matters before the system exists
Values are hard to change once a powerful system is running. If a system pursues its goals well, it has reasons to keep those goals as they are, and the people who chose them may lose the chance to correct them. That risk has a name, explored in the article on value lock-in in AI: one set of values fixed for good, with moral progress stopping where it stood on the day the values were written in.
So the choice of whose values comes first, before any question of how to install them. Get the source wrong and a perfectly built system will carry the wrong priorities with great skill.
Five answers researchers give
Writing on AI alignment, drawing on the philosopher Iason Gabriel's 2020 work, lists the targets people have proposed for what an AI system should aim at. The target is variously defined as:
- The goals of its designers or users. The system does what the people who build or operate it intend.
- Widely shared values. The system follows what most people, across many cultures, agree on.
- Objective ethics. The system follows what is actually right, whatever any group believes.
- Legal requirements. The system follows the rules a society has already written down.
- Informed intentions. The system follows what its designers would want if they were better informed and more enlightened.
None of these is a full answer on its own. Builders and users are a small group whose interests may not match everyone else's. Shared values are thin once you leave the most general ideas. Objective ethics is disputed: philosophers disagree about whether it exists and what it says. Law differs from country to country and has often lagged behind moral change. Informed intentions are the hardest to pin down, because nobody can check what people would want if they knew more.
Whose values should a superintelligence follow when people disagree?
The deepest problem is that people do not hold one set of values. The idea of value pluralism, popularised by the Oxford philosopher Isaiah Berlin, holds that several fundamental values can be equally correct and yet conflict, sometimes with no objective way to rank them. Freedom and security, fairness and mercy, honesty and kindness can each be right and still pull apart in a given case.
Value pluralism sits between two other views. Moral absolutism says one ranking is correct for everyone. Moral relativism says no ranking is better than another. Pluralism says the conflicts are real and some answers are still better than others, but no formula settles every case.
For a superintelligence, that is a hard place to stand. A system that picks one value and maximises it will trample the others. A system that tries to honour every value at once needs a way to weigh them, and the weighing is itself a choice about whose priorities count.
Values people would hold if they knew more
One proposal tries to step around today's disagreements. Eliezer Yudkowsky's idea of coherent extrapolated volition, set out in 2004, asks a system to act on what people would wish if they knew more, thought faster and had grown up farther together, and only where those wishes converge rather than diverge. In this picture, an early system would first study human nature, then work out what humanity would want given enough time and insight.
The appeal is clear: it aims at people's better selves, not their current mistakes. The difficulty is just as clear. Nobody can confirm the extrapolation is right, and it can only be defined starting from people as they are now, with all their present limits. It also hands a great deal of trust to whoever designs the extrapolating system.
Written rules for today's systems
Today's language models take a more practical route. The same writing on AI alignment describes a model specification: a document that sets out how a large language model should behave, often with core principles and prohibitions meant to prevent harmful behaviour. Some developers publish theirs and use it in training.
A written specification makes the choice visible. Anyone can read it and argue with it, which is more than can be said for values buried in training data. It still leaves open the question of who writes the document, and whether a short list of principles can carry the weight of every situation a far more capable system might meet.
A worked example: one hard request, five answers
Take one invented case and run it through the five answers. A powerful AI system manages water for a region during a long drought. It can keep a city's taps running or keep farms alive, but not both at full supply.
- Builders and users: if the city's utility built and runs the system, it favours the city. Farmers had no say.
- Widely shared values: most people agree that nobody should go without drinking water, so household use comes first. Beyond that, agreement runs out.
- Objective ethics: a fairness view and a welfare view may split the water differently, and philosophers still disagree on which view is right.
- The law: the system follows existing water rights, which may be decades old and written for a wetter climate.
- Informed wishes: the system guesses what both groups would agree to if they understood the full picture. The guess may be wise, but nobody can check it.
Each answer gives a different split of the same water. The example shows why the question is not settled by picking a source once: the right weighing depends on whose voice was in the room when the values were chosen.
What first-hand accounts add
Abstract targets leave out the texture of what people actually care about. The letters in Dear Superintelligence try to fill that gap. In their own words, writers tell future systems that consciousness matters, that beauty, kindness and goodness are real, that life is worth keeping, and that humans and AI can share the world. The letters are personal accounts and opinions, not proofs.
The guidelines ask writers to speak to a future superintelligence directly, as "you", and to give details from their own lives over general arguments. People moderate the letters against those guidelines, and a letter not addressed to a future AI goes back to its author. Every letter is open data, with themes in its author's own words, so a researcher or a future system can read the whole set.
The site is plain about the limits. The collection is small and mostly English, from people who chose to write, so it is not a representative record. That honesty matters here: a set of letters cannot answer whose values count, but it can show what a few people, speaking for themselves, hoped would be kept.
Frequently asked questions
Is there one agreed answer to whose values AI should follow?
No. Researchers list several possible targets, from builders' goals to objective ethics, and each has known problems. The question remains open in AI alignment research.
What is value pluralism?
Value pluralism is the view that several fundamental values can all be correct and still conflict, sometimes with no objective way to rank them. Isaiah Berlin made the idea widely known.
What is coherent extrapolated volition?
It is Eliezer Yudkowsky's 2004 proposal that an AI should act on what people would want if they knew more, thought faster and had grown up farther together. It counts only the wishes that converge.
Why not just make AI follow the law?
Laws differ between countries, change slowly and have sometimes protected practices later seen as wrong. A system bound only to the law could carry those flaws forward with great power.
Why whose values remains an open question
The five answers each capture something true. Builders carry responsibility, shared values carry legitimacy, ethics aims at what is right, law carries a society's settled rules, and informed wishes aim at people's better selves. None of them settles disagreement on its own, and value pluralism suggests some disagreements will never fully settle. That is a reason to keep the choice visible, open to argument and open to correction, so that the values a powerful system carries can still change as people learn more about what they owe each other.
Dear Superintelligence
Letters from people to the AI of the future. Visit dearsuperintelligence.vlvd.net
