zappz

All topics

reward model

2 articles

A reward model is a machine learning model that takes an AI system's output and returns a number estimating how much people would prefer it.

Reward models exist because many qualities people care about are hard to write down as a rule. Nobody can code a formula for a helpful answer or a clear summary, but people can readily say which of two answers they like better. A reward model is trained on many such human comparisons. Shown a prompt and a response, it learns to give higher scores to the kind of response people tended to choose.

Its main use is in reinforcement learning from human feedback. First, people compare pairs of outputs. Second, a reward model is trained to predict those choices. Third, the main model is tuned to produce outputs the reward model scores highly. The reward model stands in for the human judges, which makes it possible to score far more outputs than people could ever read. The idea was shown on simulated robots and video games before being applied to language models.

Reward models shape how chat assistants behave, so their flaws carry weight. The score is a learned approximation of preference, not preference itself. It inherits the biases and inconsistencies of the people who supplied the comparisons, and a model tuned against it can learn to exploit its blind spots, producing answers that score well without being better, such as flattering or confident-sounding ones. Understanding the reward model is therefore central to understanding why an AI system acts as it does.

  1. reward modelWhat Is a Reward Model in Machine Learning? Scoring TasteA reward model reads a prompt and an answer and returns one number that predicts how much people would prefer it. It is trained on human choices between pairs. · 8 min read
  2. rlhfWhat Is RLHF and How Does It Work? Training on PreferencesRLHF trains a reward model on human rankings of answers, then tunes the language model against it. The three steps, one example and where it falls short. · 8 min read

Related topics

All topics