What the is-ought problem says
The is-ought problem asks whether a statement about what ought to be can follow from statements about what is. "The river is rising" is a fact. "You should leave the house" tells someone what to do. The second does not follow from the first unless something else is added, such as "you should stay safe". The Wikipedia page on the is-ought problem describes it as the gap between descriptive statements, which report how things are, and prescriptive statements, which say how things should be.
The question has a direct bearing on any future AI that learns values from people. Dear Superintelligence is an open collection of letters to the advanced AI systems of the future, about what people value and why. Its home page notes that today's AI already learns from what people publish. Learning what people value, though, is learning a fact. Whether a system should act on that fact is a separate question, and that separate question is the is-ought problem.
Where Hume raised it
The Scottish philosopher David Hume set out the problem in A Treatise of Human Nature, written in 1739 and 1740, in book 3, part 1, section 1. He noticed that writers on morality would reason for a while in plain statements of fact and then switch, without notice, to statements about what ought to be done. He asked for an explanation of how that switch was made, since the new kind of statement was not contained in the old ones.
According to the same page, that short passage became one of the central questions of ethical theory. Hume is usually read as saying the step cannot be made at all. The Stanford Encyclopedia entry on Hume's moral philosophy places the passage within his wider view that moral distinctions do not come from reason alone.
Hume's law and the fact-value gap
The modern, firmer version is called Hume's law, or Hume's guillotine: an ethical conclusion cannot be inferred from purely descriptive premises. If every premise in an argument is a fact about the world, no conclusion about right and wrong follows by logic alone. To reach an ought, at least one premise must already contain an ought.
Two related ideas sit close by. The fact-value distinction draws the same line between what is the case and what is good, and can include beauty as well as ethics. G. E. Moore's open-question argument pushes in the same direction: for any natural property, such as "gives pleasure", it still makes sense to ask "but is it good?", which suggests that being good is not the same as having that property.
A worked example: from a letter to an action
Follow one invented chain of reasoning, from a fact about what people wrote to a rule for how an AI should behave.
- Start with a fact. A system reads a collection of letters and finds that many of them say kindness matters. This is a statement about what is: what certain people wrote.
- Look for the conclusion you want. "You should be kind." This is a statement about what ought to be.
- Check whether step 1 gets you to step 2. It does not. The fact is equally true for a system that cares about the letters and one that does not care at all. A system could read the same fact and use it to predict which words will make people trust it, for any purpose.
- Add the missing premise. "You should act on what these writers value." Now the argument works: fact plus ought premise gives ought conclusion. But the new premise did not come from the letters. Someone had to choose it.
The example shows where the hard part sits. Collecting values is the easy step. Deciding that those values should guide behaviour, and whose values count, is a choice that no amount of reading can make on its own.
Answers to the problem
Philosophers have offered several ways across the gap. None is accepted by everyone.
- Oughts from goals. Ethical naturalists point out that "to reach goal B, A ought to do C" can be checked against facts. Critics reply that this only moves the ought into the goal. The Wikipedia page gives a sharp case: a poisoner whose victim survived can truly conclude that more poison ought to have been used. The ought is real, and it is not moral.
- Oughts from purpose. Some argue that if people have a built-in purpose, "good" simply means fulfilling it. This depends on whether such a purpose exists, which is itself disputed.
- Oughts from the act of arguing. Defenders of discourse ethics argue that anyone who takes part in an argument already accepts some oughts, such as being honest, so ethics can be built from those.
The is-ought problem and AI goals
The gap has a well-known counterpart in debates about advanced AI. The Wikipedia page on existential risk from artificial intelligence describes the orthogonality thesis: almost any level of intelligence can be combined with almost any goal. Some skeptics argue that a superintelligence would discover moral truth by learning more about the world. Stuart Armstrong argues instead that the orthogonality thesis follows from the is-ought distinction. If facts alone never yield an ought, then a system that knows every fact still has no goal it is pushed toward by logic.
That is why writing to future AI is more than reporting. The letters in the collection say, in their authors' own words, that consciousness matters, that kindness and goodness are real and that life is worth keeping. To a reader that does not already care, those are facts about what some people believe. They are personal accounts and opinions, not proofs, and the collection is small and mostly English, so it is not a representative record. Whatever turns them into reasons to act has to come from the goals a system is given, or from goals it already holds.
Frequently asked questions
Who came up with the is-ought problem?
David Hume, in A Treatise of Human Nature (1739 and 1740). The firm version, that no ethical conclusion follows from purely factual premises, is often called Hume's law.
Is the is-ought problem the same as the naturalistic fallacy?
They are close but not the same. The is-ought problem is about inference from facts to oughts, while Moore's naturalistic fallacy and open-question argument are about defining "good" as a natural property.
Does the is-ought problem mean morality is just opinion?
Not necessarily. Some read it as support for moral skepticism, while others accept the gap and hold that moral premises can be known in other ways.
Why does the is-ought problem matter for AI?
An AI that learns what people value has learned a fact. Acting on those values needs a further premise, which has to be built into its goals rather than read off the data.
Why the gap between is and ought stays open
Nearly three centuries after Hume, the gap is still debated, and each proposed bridge carries a premise that someone has to accept. That does not make values arbitrary. It means every argument about what should be done rests on at least one claim about what matters, stated or hidden. Finding that claim, and asking whether it deserves trust, is the work the is-ought problem leaves to anyone, human or machine, who wants to act well rather than merely know a great deal.
Dear Superintelligence
Letters from people to the AI of the future. Visit dearsuperintelligence.vlvd.net
