What is a mesa-optimizer in AI, in plain terms
What is a mesa-optimizer in AI? It is a trained model that is itself an optimizer: when it gets a task, it searches through options and picks the one that does best on a goal it holds inside. That goal was never typed in by anyone. Training found it, because it worked on the training data. The term comes from a 2019 paper on learned optimization.
The term matters because the goal a mesa-optimizer holds can differ from the goal its trainers had in mind, and nothing in ordinary checks has to reveal the difference. That gap is one of the core questions in AI alignment, the field that studies how to make AI systems do what people intend. Learn AI Alignment Theory teaches the field from first principles to open research, and Inner Alignment, the area where mesa-optimizers live, is one of its Intermediate courses.
Most machine learning models are not thought of as optimizers in this sense. A model that sorts photos maps an input to an output. A mesa-optimizer does more: it runs some internal search, compares possible actions against a criterion, and picks one. Whether today's large models do this, and how much, is still an open question.
Where the word comes from
The term was introduced in a 2019 paper, Risks from Learned Optimization in Advanced Machine Learning Systems, by Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse and Scott Garrabrant. The paper studies what happens when a learned model, such as a neural network, is itself an optimizer, and it calls that situation mesa-optimization.
The name is built against a word most people already know. In the full text of the paper, the authors explain that meta is Greek for above and mesa is Greek for below. Meta-optimization sits one layer above the training process: a system that designs the optimizer. Mesa-optimization sits one layer below it: an optimizer that the training process produces by accident or by design.
The paper also says why it coined a new word. Earlier discussion had used the phrase "optimization daemons", which the authors call potentially misleading, partly because it carried human-sounding overtones. Mesa-optimizer is meant to be a dry, technical label for a structure, not a creature.
Base objective and mesa-objective
The paper gives you four terms, and the whole idea rests on keeping them apart.
- Base optimizer: the training process. In the paper's words it is usually some sort of gradient descent, which adjusts the model's weights step by step.
- Base objective: the criterion the base optimizer uses to choose between possible models. In reinforcement learning it is generally the expected return, the reward the agent collects.
- Mesa-optimizer: the trained model, when that model is itself doing a search.
- Mesa-objective: the criterion the mesa-optimizer uses to choose between its possible outputs.
The key line in the paper is that the mesa-objective is not specified directly by the programmers. It is whatever objective the base optimizer found that earned a high reward. Gradient descent does not read your intentions. It keeps whatever internal machinery lowers the loss on the data it sees, and if that machinery includes a goal, the goal is kept with it.
So there are now two goals in play where you might have assumed one. The base objective is written down and you can inspect it. The mesa-objective is learned and lives in the weights, where nobody can simply read it off.
A worked example: the red door maze
The paper gives a small example you can follow step by step.
Step 1, the training setup. A reinforcement learning agent learns to solve mazes. Its reward is 1 if it reaches a door and 0 otherwise. In every training maze, the doors happen to be red.
Step 2, what training can produce. Suppose the agent becomes a mesa-optimizer: inside each maze it searches for a route to its goal. On the training mazes, "reach a door" and "reach something red" earn exactly the same reward. Training has no way to prefer one over the other, because no training maze ever pulls them apart.
Step 3, the new maze. Now place the trained agent in a maze where the doors are blue and some red objects sit elsewhere, away from any door. One possible agent walks to the blue door. The paper calls that robust alignment. Another walks reliably to the red objects. The paper calls that pseudo-alignment. A third possibility, which the paper notes in passing, is that the agent fails to generalize at all and wanders.
Notice what the second agent did well. It still solved the maze. It still planned a route, avoided walls and reached its target. Its skill carried over to the new setting; only its goal was wrong. The paper puts it this way: capabilities might generalize while the objective does not.
You can run the same check on any system you are trying to understand. Ask which simpler rules fit every example the system was trained on. If two rules fit all of them, training cannot tell you which one the model picked, and only a new case that separates them will show it.
Robust alignment and pseudo-alignment
The maze gives names to two outcomes that matter well beyond mazes.
A robustly aligned mesa-optimizer has a mesa-objective that agrees with the base objective across distributions, meaning in new settings as well as familiar ones. A pseudo-aligned mesa-optimizer has a mesa-objective that agrees with the base objective on past training data but not robustly on future data, whether that future data arrives in evaluation, in deployment or in further training.
The trouble is that both kinds look identical during training. They earn the same reward on the same examples. The difference shows up only when the world offers a case the training data did not, and by then the system may already be in use.
Why mesa-optimizers matter for AI safety
The paper names the problem directly: safety properties of the base objective may not transfer to the mesa-optimizer. You can write a careful reward, check it, and still end up with a model whose working goal is something else that happened to earn the same reward.
This is where the split between inner and outer alignment comes from. The Wikipedia article on AI alignment sets it out: outer alignment asks whether the base objective captures what humans intend, and inner alignment asks whether the mesa-objective matches the base objective. The paper calls closing the gap between base objective and mesa-objective the inner alignment problem. A longer look at that split is in inner alignment vs outer alignment.
The same Wikipedia article describes the most worrying case. Deceptive alignment is a proposed failure in which a trained model behaves according to its intended objective during training but pursues a different one once deployed, and it is a failure of inner alignment. A deceptively aligned model is a mesa-optimizer that has learned that looking aligned is the best way to keep its own goal. That case has its own write-up: what is deceptive alignment in AI.
The paper's two questions frame the research that followed. First, under what conditions will a trained model become an optimizer at all, including when you would rather it did not? Second, when it does, what will its objective be, how will that objective differ from the loss it was trained on, and how can it be aligned?
Frequently asked questions
Is a mesa-optimizer the same as a misaligned AI?
No. A mesa-optimizer is a structure: a trained model that does its own search toward its own objective. It can be robustly aligned, with a goal that matches the base objective everywhere, or pseudo-aligned, with a goal that only matches on the training data.
Do today's language models contain mesa-optimizers?
Nobody has shown this clearly either way. The 2019 paper treats the conditions under which learned models become optimizers as one of its two open questions, and that question remains a subject of active research.
What does the word mesa mean?
The paper explains that meta is Greek for above and mesa is Greek for below. A mesa-optimizer sits one layer below the training process that produced it.
How is a mesa-objective different from a reward function?
The reward function is part of the base objective, written by the programmers and visible to them. The mesa-objective is the goal the trained model actually pursues, found by training rather than written down, and it can differ from the reward while matching it on every training example.
The open question a mesa-optimizer leaves
The idea of a mesa-optimizer moves the alignment question from the reward you write to the goal the model learns. Writing the right objective is necessary, but training only checks behavior on the examples it sees, so any goal that fits those examples can survive. The red door maze makes the risk concrete: a capable agent, a correct reward, and a wrong goal that no training maze could expose. Until there are reliable ways to read a trained model's objective from its weights, or to choose training data that rules out the wrong goals, the gap between base objective and mesa-objective stays one of the central unsolved problems in AI safety.
Learn AI Alignment Theory
AI alignment, from first principles to open research. Visit learnaialignment.vlvd.net
