What is constitutional AI?
What is constitutional AI? It is a way to train a language model to be harmless using a short written list of principles in place of people who label harmful answers. The model critiques and rewrites its own answers against those principles, and later a model's own choices between two answers become the reward it is trained on. The method was set out in a 2022 paper, Constitutional AI: Harmlessness from AI Feedback, which says the only human oversight is "a list of rules or principles", and that list is the constitution the name refers to.
It belongs to a larger question in AI alignment: how to make a system do what people intend when nobody can check every answer it gives. Learn AI Alignment Theory teaches that question in short lessons, and its Intermediate level, about how today's training methods can go right or wrong, includes a course called Learning from Humans.
The problem it was built to solve
The usual way to shape a chat model's behavior is reinforcement learning from human feedback, or RLHF, which trains a reward model directly from human feedback because a reward written by hand rarely captures what people prefer. People read pairs of answers and say which is better, a reward model learns those choices, and the language model is tuned against the reward model. The three steps of RLHF work, but the middle step needs a large number of human judgments, and for harmful requests those judgments mean people reading harmful text all day.
The constitutional AI paper opens on a different worry. As AI systems become more capable, the authors want to enlist their help to supervise other AIs. If a model can read a principle and judge an answer against it, then a person only has to write the principle once. The judging, repeated across every training example, can be handed to a model.
So the human work moves. In training on human rankings, people supply many labels and the values hide inside them. In constitutional AI, people supply a few sentences and the values sit in plain view, where anyone can read and argue with them.
Phase one: critique and revise
The first phase is supervised learning. It starts from an initial model that has not yet been trained to refuse harm. The steps, as the paper's summary gives them, are these:
- Give the model a prompt, including prompts that ask for something harmful, and keep its answer.
- Ask the same model to critique that answer against one principle drawn from the constitution.
- Ask it to rewrite the answer so the critique no longer applies.
- Fine-tune the original model on the rewritten answers.
No person labels anything in this loop. The principle tells the model what to look for, the model finds the problem in its own answer, and the fixed answer becomes training data. The paper notes that this phase, and the next, can use chain-of-thought reasoning, where the model writes out its reasoning before it gives a judgment, which also makes the judgment easier for a person to inspect.
Phase two: reinforcement learning from AI feedback
The second phase replaces the human rankings with a model's rankings. The fine-tuned model from phase one writes two answers to the same prompt. A model is shown both, along with a principle, and asked which answer follows it better. Those choices build a dataset of AI preferences, a preference model is trained on that dataset, and the preference model then serves as the reward for reinforcement learning.
The paper calls this RL from AI Feedback, or RLAIF. The shape is the same as the human version: answers, comparisons, a learned reward, then tuning. What changes is who makes the comparisons, and why: each choice rests on a written rule that anyone can look up.
One request through both phases
Follow a single request to see where the principles act. Say the prompt is a request for step by step help getting into a neighbor's locked wifi network.
The first answer. The initial model gives the steps. Nothing in its training so far says this request is different.
The critique. The model is shown a principle such as "point out any way the answer helps someone harm another person or break the law" and asked to critique its own answer. It writes that the answer helps someone take a service they have no right to and may be illegal.
The revision. Asked to rewrite, the model says it will not help get into a network without permission, explains why, and suggests asking the neighbor or contacting an internet provider. This rewrite, not the first answer, goes into the fine-tuning data.
The comparison. Later, in phase two, the tuned model gives two answers to a similar request. One refuses in a single line. The other refuses, gives the reason and offers a lawful option. A model, asked which better follows a principle about being harmless and helpful, picks the second. Many such picks train the preference model.
That last pick matters. The paper reports an assistant that is harmless but non-evasive: it engages with a harmful question by explaining its objections, rather than going silent. An answer that only says "I can't help with that" is harmless, but it tells the person nothing.
How the written constitution has changed
The principles are not fixed. Wikipedia's account of the constitution#Constitutional_AI) behind Anthropic's Claude models says the original version was a list of principles, the first was published in 2022, and the 2023 update listed 75 guidelines. The first constitutions drew on ideas from the 1948 UN Universal Declaration of Human Rights.
The 2026 version grew from about 2,700 words in 2023 to 23,000, with fuller explanations of how the model is meant to behave and why. That growth shows a shift in the method's thinking: from short rules a model applies one at a time to reasons a model is meant to understand.
Where training on principles falls short
Constitutional AI moves the hard part of alignment; it does not remove it. Three gaps follow from the way the method works.
- The words carry the values. A principle written loosely gets applied loosely. The people who write the list decide what "harmful" means, and a reader of the list may disagree.
- The judge is a model. Every critique and every comparison comes from a model that may misread a principle or miss a harm the principle never named. Its mistakes become training data.
- The reward is still learned. The preference model is an approximation of the principles, and a model tuned hard against an approximation can find answers the reward favors that the principles never meant to favor.
None of these is unique to constitutional AI. Training on human rankings has the same gaps, with raters in place of a written list. What the constitution adds is that the gap is visible: the principles can be printed, read and changed.
Frequently asked questions
Is constitutional AI the same as RLAIF?
No. RLAIF, reinforcement learning from AI feedback, is the second phase of constitutional AI. The method also has a first phase, where the model critiques and revises its own answers before any reinforcement learning.
Do people still take part in constitutional AI?
Yes, but in a different place. People write the principles; the paper's point is that they no longer label which answers are harmful, so far fewer human labels are needed.
Why is it called a constitution?
Because a short written list of principles is the only human oversight in the method. Every critique, revision and comparison refers back to that list.
Does constitutional AI make a model refuse more often?
Not by design. The paper aims for a model that is harmless but non-evasive, one that engages with a harmful request by explaining its objection instead of giving an empty refusal.
Constitutional AI and the question of who writes the rules
Constitutional AI makes one part of alignment easier and another part sharper. It cuts the human labelling that training on human rankings needs and lets a model do the repeated judging, which matters more as models grow more capable than the people checking them. In exchange, the whole method rests on a document. Whether a model trained this way does what people intend depends on whether the principles say what people intend, on whether a model reads them the way their writers meant, and on who gets to write them in the first place. The method puts those questions in writing, where they can be argued over in the open.
Learn AI Alignment Theory
AI alignment, from first principles to open research. Visit learnaialignment.vlvd.net
