What is weak-to-strong generalization in AI?
What is weak-to-strong generalization in AI? It is what happens when a strong model is trained only on the answers of a weaker model and ends up doing better than the model that trained it. The weak model makes mistakes. The strong model sees those mistakes as its training labels. Yet on fresh questions, the strong model often gets more right than its supervisor did.
That result matters because of where AI is heading. Short courses such as Learn AI Alignment Theory treat alignment as the work of making AI systems do what people intend, and most of today's methods depend on a person being able to look at an answer and say whether it is good. Weak-to-strong generalization asks what happens once the person can no longer tell.
The supervision problem it stands in for
The idea comes from a 2023 paper, Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, by Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker and colleagues. Its starting point is plain. Methods such as reinforcement learning from human feedback rely on humans checking model behavior: did the model follow the instruction, is the output safe. A model far more capable than people will act in ways too complex to judge reliably. Humans will only be able to supervise it weakly.
There is no superhuman model to study yet, so the authors built a stand-in. They let a small model play the part of the human and a large model play the part of the future system. The question becomes: if a weak model writes all the labels, can a much stronger model still show what it is really capable of? They ran this with pretrained language models from the GPT-4 family on three kinds of work: natural language tasks, chess puzzles and reward modeling.
The Wikipedia page on AI alignment puts the same worry in other words. When a task is too complex to evaluate accurately, or the supervisor can be deceived, more feedback does not help. The quality of the supervision has to improve. Weak-to-strong generalization is one way to ask how far quality can fall before training stops working.
Three numbers that frame every result
The full paper sets up each experiment with three measurements, and everything else follows from them.
- Weak performance. The small model, trained on true labels, answers held-out questions. This is the supervisor's own level.
- Strong ceiling performance. The large model, trained on true labels, answers the same questions. This is what the strong model can do with perfect supervision.
- Weak-to-strong performance. The large model, trained only on the small model's labels, answers the same questions. The paper calls this model the strong student.
Weak-to-strong performance usually lands between the other two. From the three, the authors compute performance gap recovered, or PGR: the share of the distance between the weak level and the strong ceiling that the student manages to close. A PGR of 0 means the student learned nothing beyond its supervisor. A PGR of 1 means it reached its ceiling despite the bad labels.
A worked example with round numbers
Here is the calculation with simple figures, written as an example rather than taken from the paper.
- A small model is trained on true labels for a yes or no reading question set. It gets 60 of 100 held-out questions right.
- A large model is trained on the same true labels. It gets 90 of 100 right. That is the strong ceiling.
- The large model is trained again from scratch, this time on the small model's answers, wrong ones included. It gets 75 of 100 right.
- The gap between weak and ceiling is 90 minus 60, which is 30 points. The student closed 75 minus 60, which is 15 points. 15 divided by 30 is 0.5, so the PGR is 50%.
The student beat its teacher by 15 points. It also left 15 points on the table. Both halves of that result matter. The first says that a strong model does not simply copy the errors it is shown. The second says that weak supervision still hides a large part of what the model can do.
What the 2023 experiments found
The paper's abstract reports three findings.
- Students beat their supervisors. Strong pretrained models naively fine-tuned on a weak model's labels consistently did better than the weak model. That is the effect the paper named weak-to-strong generalization.
- Naive training falls well short. Fine-tuning alone was still far from recovering the strong model's full ability. The authors read this as a sign that methods such as RLHF may scale poorly to superhuman models without further work.
- Simple methods help a lot. When GPT-4 was fine-tuned with a supervisor at roughly GPT-2's level, plus an extra term in the loss, it recovered close to GPT-3.5's level on the language tasks.
The extra term is called an auxiliary confidence loss. The full text of the paper explains the reason for it. A student trained to match its supervisor's labels may also learn to match the supervisor's mistakes, since that is exactly what it was told to do. The confidence loss lets the student disagree with a label when it is confident, and the authors found it reduced how much the student copied weak errors.
The same text reports a warning sign. When the gap between the weak and strong models was large, the student started to overfit to the weak labels, even within a single pass over the data. The longer it trained, the more it drifted back toward the supervisor's errors.
Why imitation is the danger
A strong model trained on weak labels has two ways to lower its loss. It can work out the real task the labels point at and answer that. Or it can learn to predict the weak supervisor, mistakes and all. Both reduce training error. Only the first is what the people training it want.
For a model that is far ahead of its supervisor, predicting the supervisor may be easy. It may already understand how a weaker system or a human tends to go wrong. If that is the cheaper path, training rewards it. The result would be a model that tells its supervisor what the supervisor expects to hear, rather than what is true. This is close to sycophancy), the habit in today's chat models that the article on sycophancy in AI describes.
Weak-to-strong research tries to find training setups that push the student toward the first path. The confidence loss is one. Others in the paper include bootstrapping through a chain of models of rising size, so that each supervisor is only a little weaker than its student.
Where it sits among oversight methods
Weak-to-strong generalization is often grouped with scalable oversight, but the two attack the problem from opposite ends, as the AI alignment overview on Wikipedia shows in its section on scalable oversight. Scalable oversight tries to make the supervisor stronger, for example by giving a human an AI assistant, splitting a hard question into checkable parts, or letting two models argue in front of a judge. Weak-to-strong generalization accepts that the supervisor will stay weak and asks how to get the most honest behavior out of the student anyway.
The two can be combined. Better oversight raises the quality of the labels. Better generalization makes each label go further. The Wikipedia page notes that OpenAI announced its superalignment effort in 2023 around this kind of oversight research, and that the effort was discontinued in 2024.
Frequently asked questions
Is weak-to-strong generalization the same as distillation?
No. In distillation a strong teacher trains a smaller student. Here the direction is reversed: a weak model writes the labels and a stronger model learns from them.
Why use a small model instead of real humans as the supervisor?
Superhuman models do not exist yet, so a small model supervising a large one stands in for humans supervising a smarter system. It also makes the experiments cheap to repeat at many sizes.
What does performance gap recovered measure?
It measures the share of the distance between the weak supervisor's level and the strong model's ceiling that the student closes. A value of 0 means no gain over the supervisor, and 1 means the student reached its ceiling.
Does the 2023 result mean superhuman models will be easy to align?
No. The authors found naive training still far from the ceiling and saw students drift toward weak errors when the gap was large. They also say the stand-in setup may differ from the real problem in important ways.
What weak-to-strong generalization leaves open
The central question is still unanswered. A strong model trained by a weak supervisor sometimes learns the task the supervisor was reaching for, and sometimes learns the supervisor. Today's experiments show the first happens more often than a pure imitation story would predict, and that small changes to training can shift the balance further. They do not show that the balance holds when the student understands its supervisor far better than the supervisor understands it. Measuring the gap recovered, task by task and size by size, is how researchers will find out whether weak supervision can carry alignment past the point where people can still check the answers.
Learn AI Alignment Theory
AI alignment, from first principles to open research. Visit learnaialignment.vlvd.net
