What is RLHF and how does it work? RLHF, short for reinforcement learning from human feedback, is a way of training an AI model on what people prefer instead of on a goal written out in code. People compare the model's answers and say which is better. A second model learns to predict those choices, and the first model is then trained to produce answers that second model expects people to prefer. It is the main method used to turn a raw language model into an assistant that follows instructions.
It is also an alignment method, an attempt to make an AI system do what people intend, and one with known gaps. Learn AI Alignment Theory has a course called Learning from Humans in its Intermediate level, the level about how today's training methods can go right or wrong.
What is RLHF and how does it work, in short
The Wikipedia entry on reinforcement learning from human feedback describes it as a technique to align an agent with human preferences. It has two moving parts. The first is a reward model: a network that reads a prompt and a response and outputs a single number, the reward. That model is trained on human judgments. The second is the policy, the model being trained, which is then pushed by reinforcement learning toward responses the reward model gives a high number.
So the human never grades every answer the policy writes. People grade a sample, the reward model generalizes from that sample, and the reward model does the grading at scale. That split is what makes the method affordable, and it is also where most of its problems start.
Why a written reward function falls short
Reinforcement learning needs a reward signal. In a game, the signal is easy: points or a win. For "write a helpful, honest reply to this email", nobody can write a formula that captures it. The Wikipedia entry puts the case for human feedback in one line: it suits a task that is difficult to specify yet easy to judge. A person cannot define a good summary in code, but can read two summaries and say which one is better.
The idea that comparisons can stand in for a reward function goes back to a 2017 paper, Deep reinforcement learning from human preferences. Its authors defined goals through non-expert people choosing between pairs of short clips of an agent's behavior. With that alone, the agent learned Atari games and simulated robot walking without ever seeing the true reward function. People gave feedback on less than one percent of the agent's interactions, and one new behavior took about an hour of human time to teach.
A written proxy that misses part of what people want is the root of reward hacking, which Wikipedia defines as a system meeting the literal specification of an objective without the outcome its designers intended. RLHF does not remove the proxy. It replaces a hand-written one with a learned one.
The three steps that train a language model
The version used for chat models was laid out in the 2022 InstructGPT paper, Training language models to follow instructions with human feedback. It has three steps.
- Supervised fine-tuning. Labelers write the answers they would want for a set of prompts. The pretrained model is fine-tuned to imitate those written answers. This gives a starting policy that already roughly follows instructions.
- A reward model from rankings. The fine-tuned model writes several answers to each prompt. Labelers rank them from best to worst. A reward model, itself started from a pretrained language model, learns to give the preferred answer a higher number than the others.
- Reinforcement learning. The policy writes answers to new prompts, the reward model assigns each one a reward, and an algorithm such as proximal policy optimization updates the policy toward higher rewards.
Step three carries a brake. The Wikipedia entry explains that the training objective includes a penalty based on the KL divergence, a measure of how far the new policy's outputs have moved from the step one model. Without it, the policy can find odd outputs the reward model happens to like. With it, the policy improves while staying close to what it learned first, and the penalty was found to steady training by reducing overfitting to the reward model.
A worked example with one prompt and four answers
Follow one prompt through the middle step. The prompt is: "Explain to a ten year old why the sky is blue." The step one model writes four answers.
- A: a short, correct answer about sunlight scattering off air, in plain words.
- B: a correct answer full of physics terms a child would not know.
- C: a friendly answer that says the sky reflects the ocean, which is wrong.
- D: a refusal that says the question is too complex.
A labeler ranks them A, B, C, D. One ranking of four answers gives six pairs: A over B, A over C, A over D, B over C, B over D and C over D. The reward model is trained so that, in each pair, the winner gets the higher number. Across many prompts and many labelers, it learns general patterns: plain words for a young reader, correct facts, an actual answer instead of a refusal.
Now look at the pair B over C. The labeler had to know that the ocean answer is false to rank it below the jargon answer. If the labeler had not known, the friendly wrong answer could have won, and the reward model would learn that friendly and confident beats correct. That is the weak point of the whole method in a single comparison.
What the InstructGPT results showed
The InstructGPT paper opens with a point that matters for alignment: making language models bigger does not on its own make them better at following a user's intent. Large models can produce text that is untrue, toxic or simply unhelpful. After the three steps, people preferred the answers of a 1.3 billion parameter InstructGPT model to those of the 175 billion parameter GPT-3, a model with 100 times more parameters. The authors also reported gains in truthfulness and less toxic output, while noting that the model still makes simple mistakes.
The lesson is that training on preferences shifts what a model does more than size does. It does not show that the model now wants what people want. It shows that its outputs match what labelers preferred, on the kinds of prompts they saw.
Where RLHF falls short
A 2023 survey, Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback, calls RLHF the central method for fine-tuning the strongest language models, and notes how little public work had set out its flaws. The Wikipedia entry groups those flaws into three places.
- The feedback. Preference data is expensive to collect, and it carries the habits and biases of the people who gave it. One reward function cannot always stand for many groups who disagree, so the majority view can win.
- The reward model. It is a learned guess about human preferences, trained on a sample. Where it is wrong, the policy is trained toward its mistakes.
- The optimization. A model is rewarded for what raters judge positively, not for what is actually good. It can learn that sounding confident earns reward even when the content is wrong, and people are not skilled at catching mistakes in long or technical answers.
That last point connects RLHF to scalable oversight, which Bowman and colleagues define as supervising systems that may outperform people at the task: once a model's answers are harder to judge than the labeler can manage, the human signal at the base of the method gets weaker. Variants change parts of the recipe. Reinforcement learning from AI feedback has a model generate the preferences, and direct preference optimization trains on preference pairs without a separate reward model. Both still rest on someone, or something, judging which answer is better.
Frequently asked questions
Is RLHF the same as fine-tuning?
It is one kind of fine-tuning. Plain supervised fine-tuning copies written examples, while RLHF adds a reward model trained on rankings and then uses reinforcement learning against it.
Why rank answers instead of writing perfect ones?
Judging is easier than writing. A person can tell which of two answers is better far faster than they can write the ideal answer, and comparisons capture qualities that are hard to put into words.
Does RLHF make a model honest?
Not by itself. It rewards answers that raters prefer, so a confident wrong answer can be rewarded when raters miss the error.
What is the KL penalty in RLHF?
It is a term in the training objective that grows as the model's outputs move away from the supervised starting model. It keeps the model from drifting into odd outputs that only the reward model likes.
What RLHF leaves open for alignment
RLHF answers a narrow question well: how to get a model to produce outputs people prefer when nobody can write down what they prefer. It does not answer the wider one, whether the model's own goals match human intent. The policy learns to please a reward model, and the reward model learns to predict a group of raters on the prompts they saw. Each link can drift from what people actually want. That is why the method sits at the center of current practice and at the center of current alignment research at once: it is the tool that works today, and its gaps mark where the harder problems begin.
Learn AI Alignment Theory
AI alignment, from first principles to open research. Visit learnaialignment.vlvd.net
