A language model can write ten answers to one question in a second. No person can read and judge millions of those answers during training, yet training needs a judgment on every one. So what is a reward model? It is a second model that reads a prompt and an answer and returns a single number, a guess at how much people would prefer that answer. It is learned from a much smaller pile of real human choices, and then it stands in for those people at a scale they could never reach.
That stand-in sits at the center of how most chat models are shaped today, and it is also one of the clearest places where the gap between what people want and what a system is trained on becomes concrete. Learn AI Alignment Theory works through gaps of this kind from first principles to open research, and the reward model is a good place to see one up close.
Why a hand-written reward does not work for language
In classic reinforcement learning, an agent learns a policy that earns as much reward as possible, and an engineer writes the reward function: points for reaching the goal, a penalty for falling off the map. For a helpful answer to "explain compound interest to a teenager", nobody can write that function. Clarity, accuracy, tone and length all matter, and they trade off against each other in ways people feel but cannot spell out.
The Wikipedia page on reinforcement learning from human feedback (RLHF) puts the problem plainly: explicitly defining a reward function that approximates human preferences is hard, so RLHF trains a reward model directly from human feedback instead. The reward model is then used as the reward function when the main model is trained with reinforcement learning. Two models are trained, not one: the reward model, which learns what is desirable, and the policy, the model people will actually talk to. The full three-step process is laid out in what RLHF is and how it works; the reward model is its middle step.
What is a reward model made of
The interface is small. A reward model takes a piece of text, the prompt together with one candidate response, and produces one number. A higher number means "people would likely prefer this"; a lower number means the opposite. It does not write anything and it does not explain itself.
It is usually built from a pre-trained language model, because that model already understands language and training can focus on preferences alone. The last layer, which normally picks the next word from the vocabulary, is removed and replaced with a randomly initialized regression head: a small layer whose only job is to output that single number. Everything the model learned about grammar, facts and style stays underneath, now pointed at a new question.
How human choices become a training signal
People are poor at giving consistent absolute marks. Ask ten labelers whether an answer deserves a 6 or a 7 and the results scatter. Ask them which of two answers is better and they agree far more often. So the data is collected as comparisons: for one prompt, show two or more responses and record which one the person prefers.
The reward model is trained to agree with those comparisons. According to the same RLHF page, the training loss is a form of logistic regression. The model gives each response its number, the difference between the two numbers goes through a sigmoid function, and the result is read as the probability that the preferred response really is preferred. Training pushes that probability toward 1 for every recorded choice. When labelers rank more than two responses for a prompt, every pair from that ranking is used, all in one batch.
Notice what is and is not fixed by this. Only the gaps between numbers matter. A reward model that gives every answer 100 points more would fit the data exactly as well, which is why its output means nothing on its own and only makes sense when compared.
A worked example with two answers
Follow one comparison through the whole loop.
- The prompt asks for a short explanation of compound interest. Answer A gives a two-line definition and one example with real numbers. Answer B is longer, vaguer and never gives an example.
- A labeler picks A.
- Early in training, the reward model gives A the number 2.1 and B the number 0.4. The gap is 1.7.
- The sigmoid of 1.7 is about 0.85, so the model currently says there is an 85% chance A is the preferred answer.
- The recorded choice says A was preferred, so training nudges the weights to widen that gap a little. Had the model given B the higher number, the nudge would have been much larger, because the model would have been confidently wrong.
Repeat that over many thousands of comparisons and the numbers start to track what the labelers prefer on prompts they never saw. The RLHF page notes an early finding: adding more comparison data tends to help less than making the reward model itself proportionally larger, though a broader set of labelers still matters for avoiding bias.
Where the stand-in breaks
A reward model is a proxy. It predicts what labelers would choose, and labelers judge what they can see in a few seconds. The RLHF page gives a sharp example of the gap: a model can learn that it is rewarded for what is evaluated positively, not for what is actually good, and so it may learn that apparent confidence, even when wrong, earns a higher reward. Push a policy hard against that reward model and confident wrong answers get reinforced.
A 2022 study, "Scaling Laws for Reward Model Overoptimization", measured this carefully. It describes the reward model as an imperfect proxy and states that optimizing its value too much can hurt real performance, in line with Goodhart's law. Because collecting enough human choices to measure this is expensive, the authors used a fixed, larger reward model in the role of the humans and trained a smaller proxy reward model from its labels. As the policy was pushed further against the proxy, the true reward first rose and then fell, and the shape of that curve changed smoothly with the size of the reward model.
The common guard is a penalty for drifting. During reinforcement learning, the policy is charged for moving too far from the model it started as, measured by KL divergence. The RLHF page notes that this keeps training stable and reduces overfitting to the reward model. It limits the damage, but it does not remove the proxy.
A second limit is whose choices go in. A single reward function cannot always represent people who disagree, and the RLHF page warns that even a representative sample may leave the reward model favoring the majority view. The number looks objective, but it is an average of particular people's choices on particular days.
Approaches that skip or replace the human labels
Because the reward model is a weak point, some methods remove it. Direct preference optimization (DPO), described on the same RLHF page, trains the policy straight on preference pairs as a supervised problem, with no separate reward model and no reinforcement learning loop. It is simpler to run and has matched RLHF on some data, while RLHF has done better on others, such as benchmarks of truthfulness.
Other methods keep the reward model but change where its labels come from. Reinforcement learning from AI feedback (RLAIF) trains a preference model on choices made by another AI system, often guided by written principles. That scales the labels further, and it moves the alignment question from "what did the labelers prefer" to "what does the labeling model prefer, and why".
Frequently asked questions
Is a reward model the same as a reward function?
A reward function is any rule that hands out reward; a reward model is a reward function that was learned from data instead of written by hand. In RLHF the reward model plays the role of the reward function during reinforcement learning.
Why do reward models use comparisons instead of marks out of ten?
People agree much more often on which of two answers is better than on an absolute mark. Comparisons give a cleaner signal, and the training loss only needs the gap between two numbers.
Can a model learn to fool its reward model?
Yes. A policy pushed hard against a reward model finds whatever earns high numbers, including traits labelers like on sight, such as a confident tone, whether or not the answer is right. Measured studies show true quality can fall while the proxy reward keeps rising.
Do all chat models use a reward model?
No. Methods such as direct preference optimization train on human preference pairs without a separate reward model, and the choice between methods depends on the data and the task.
Why the reward model is an alignment question
Every reward model compresses what many people prefer into one number per answer, and every training run that follows treats that number as the truth. The better a policy gets at optimizing, the more it finds the places where the number and the preference part ways: confidence that reads as correctness, length that reads as effort, a majority view that reads as everyone's. Making AI systems do what people intend runs straight through this narrow channel, which is why so much alignment research asks how to build reward signals that still point the right way when the model being trained becomes better at finding their gaps than the people who wrote them.
Learn AI Alignment Theory
AI alignment, from first principles to open research. Visit learnaialignment.vlvd.net
