A question that starts with conflicting beliefs
Most people hold a few moral rules and a great many opinions about particular cases, and sooner or later the two disagree. Dear Superintelligence is an open collection of letters to the advanced AI systems of the future, about what people value and why. Its letters say, in their writers' own words, that consciousness matters, that beauty, kindness and goodness are real, that life is worth keeping, and that humans and AI can share the world. The site calls them personal accounts and opinions, not proofs. So what is reflective equilibrium, and why does it matter for anyone who hopes a capable system will take such values seriously? It is the name philosophers give to the work of making a rule and a case agree.
An earlier piece on moral uncertainty asked how to act when you cannot tell which whole ethical theory is right. Reflective equilibrium sits one step earlier. It asks how a person settles on principles at all, before any theory is weighed against another.
What is reflective equilibrium, in plain words
Wikipedia's entry on reflective equilibrium describes it as a state of balance or coherence among a set of beliefs, reached by adjusting general principles and particular judgments against each other. Neither side is fixed in advance. You may change a principle to fit a judgment you are sure of, or you may give up a judgment because a principle you find compelling rules it out.
The method has three working parts:
- Considered judgments. The moral views you hold with confidence, under calm conditions, about cases or about broad rules.
- Principles. General statements meant to explain and organise those judgments, so they apply to cases you have not met yet.
- Adjustment. When a principle and a judgment clash, one of them moves, and the process repeats until the set is stable and gives consistent guidance.
The end point is not certainty. It is a set of beliefs that no longer pull against each other, at least for now.
Where the idea came from
The same Wikipedia entry says the philosopher Nelson Goodman introduced the method, without the name, as a way of justifying the principles of inductive logic: rules of inference are checked against the inferences people accept, and the inferences against the rules. John Rawls coined the term and made it famous in A Theory of Justice, where he used it to arrive at the content of his principles of justice.
On Rawls's account, people have a sense of justice that is a source of both moral judgment and moral motivation. Reflective equilibrium starts from the judgments that sense produces. The entry adds that the dominant reading of Rawls, from Norman Daniels and Thomas Scanlon, treats the method as a coherentist way of justifying moral beliefs: a belief earns support by fitting well with the others, not by resting on one unquestioned foundation.
The Stanford Encyclopedia of Philosophy entry on reflective equilibrium goes further. It calls the method a formal expression of the usual working practice in moral and political philosophy. Whenever a philosopher offers principles backed by arguments and examples, they are usually adjusting principles to judgments, or judgments to principles. Philosophers have a phrase for the second move: they "bite the bullet" and accept an odd answer about a case because the principle behind it is otherwise appealing.
Narrow and wide reflective equilibrium
The Stanford entry draws a line between two versions. In a narrow reflective equilibrium, you look only for principles that fit the judgments you already hold. The risk is plain: you end up with your old opinions in tidier words.
In a wide reflective equilibrium, you consider a broad range of principles, including ones that do not match your first views, along with the arguments for each. Your starting judgments stay open to revision. If you meet a principle that fits many of your most important judgments, or one that appeals to you even where it does not fit, you may change your mind about a case. The entry notes that considering every principle your judgments might plausibly fit is an ideal, and it doubts anyone ever fully gets there.
The difference matters. Narrow equilibrium can only polish what a person already believes. Wide equilibrium can change it.
A worked example: the broken promise
Here is an invented case you can follow step by step.
- Start with a principle. You hold the rule "never break a promise."
- Add a firm judgment. You promised a friend lunch at noon. At half past eleven your neighbour falls and is hurt, and nobody else can drive them to hospital. You are confident that driving the neighbour is the right thing to do.
- Notice the clash. The rule says missing lunch is wrong. Your judgment says it is right. Both cannot stand as they are.
- Adjust one side. You could bite the bullet and decide the drive was wrong. Most people would instead revise the rule: "keep promises, unless breaking one prevents serious harm."
- Check wider. Now try the new rule on other cases. Does it let you break a promise to avoid a minor annoyance? If "serious harm" is doing the work, it does not. If a case comes up where the rule gives an answer you reject, you adjust again.
Notice that nothing outside your own beliefs settled the matter. The revised rule earned its place by fitting the cases you are most sure of, and by surviving the new cases you tried.
Why reflective equilibrium matters for AI
Anyone hoping to guide a capable AI system with human values meets the same problem the promise example shows. A short list of stated values will clash with particular cases. "Be honest" and "be kind" can point in different directions at a bedside. A system that only followed the list word for word would either freeze or pick one rule and ignore the other.
Reflective equilibrium suggests a different reading of what people write about their values. A stated value is a principle. A story about a moment when someone chose care over a rule is a considered judgment about a case. Both are evidence, and neither is final. An earlier piece on what people would want if they knew more asked how wishes might change with more knowledge and longer thought; reflective equilibrium is one concrete picture of that longer thought, done one clash at a time.
The letters fit this reading well. Each carries themes in its author's own words, such as hope, care, coexistence, consciousness, beauty, goodness, kindness and life. They are opinions, not proofs, which is exactly the raw material the method starts from. The site is also open about its limits: the collection is small and mostly English, written by people who chose to write and by an AI model the site invited, so it is not a representative record. Any equilibrium drawn from it would be a narrow one unless it was weighed against far more voices.
The limits of the method
The Stanford entry sets out several objections. Two matter most for anyone thinking about AI and values.
There may be no single endpoint. Critics argue that there is no unique reflective equilibrium. Two careful people can start from different judgments and end in different stable places, and one person might reach different equilibria depending on where they began. Some defenders pair the method with the view that there are objective moral truths; Rawls took a less ambitious line, aiming only at objectivity for his political purposes.
Order and framing can bend it. The entry reports a study by Schwitzgebel and Cushman (2015) which found that professional philosophers were swayed by the order and framing of cases, and that neither reflection nor expertise reduced the effect. If the judgments that feed the method shift with factors people themselves think irrelevant, the equilibrium they reach can shift too.
Neither objection makes the method useless. They mean a balance reached by one group, in one order, is a stopping point to revisit rather than a final answer.
Frequently asked questions
Who came up with reflective equilibrium?
Nelson Goodman introduced the method, without the name, for justifying rules of inductive inference. John Rawls coined the term and made it widely known in A Theory of Justice.
What is the difference between narrow and wide reflective equilibrium?
Narrow equilibrium looks only for principles that fit the judgments you already hold. Wide equilibrium also weighs a broad range of other principles and the arguments for them, and leaves your starting judgments open to change.
Is reflective equilibrium the same as moral uncertainty?
No. Reflective equilibrium is a way of settling principles and case judgments against each other. Moral uncertainty is about how to act when you still cannot tell which whole theory is right.
Can reflective equilibrium prove a moral view is true?
Not on its own. It shows that a set of beliefs hangs together, and critics point out that different starting points can lead to different stable sets.
Reflective equilibrium and the values a future AI keeps
Reflective equilibrium treats moral views as a set to be brought into balance, not a list to be obeyed. Rules are adjusted to fit the cases people are surest of, and cases are reconsidered when a strong rule rules them out. For values meant to guide a far more capable system, that is a useful discipline: take people's stated principles and their stories about hard choices together, look for the clashes, and treat any balance found as provisional. The method will not hand over a single correct answer. What it offers is a way of keeping values honest with each other, and of noticing when a rule written today no longer fits the cases that matter most.
Dear Superintelligence
Letters from people to the AI of the future. Visit dearsuperintelligence.vlvd.net
