If you ask what is mismatched generalization in AI safety, the short answer is this: a language model learns broad skills from a huge body of text, but its safety training covers only a narrow slice of the situations where those skills can be used. Wherever capability reaches and safety training does not, a gap opens, and attackers aim at that gap. One research effort, a lab studying AI alignment through Jungian individuation, asks whether safe behavior could be grown inside a model through ordered stages of training, so that it reaches as far as the model's skills do.
What is mismatched generalization in AI safety?
The term comes from a 2023 paper by Alexander Wei, Nika Haghtalab and Jacob Steinhardt, "Jailbroken: How Does LLM Safety Training Fail?". The authors proposed two failure modes of safety training. The first is competing objectives: a model's capabilities and its safety goals conflict, so the pull to follow instructions works against the pull to refuse. The second is mismatched generalization: safety training fails to carry over to a domain where the model's capabilities still work.
The paper traces the second mode to a difference in scale. Pretraining runs on a larger and more diverse dataset than safety training, so the model ends up with many capabilities that safety training never touched. An attacker only has to find a prompt on which pretraining and instruction following generalize but safety training does not. On that prompt, the model answers, without the safety behavior it shows elsewhere.
A worked example: the Base64 request
Base64 is a binary-to-text encoding that writes data using 64 printable characters. A Base64 string reads as letters and digits with no plain words in it. The Jailbroken paper used it to show mismatched generalization in its sharpest form. Follow the steps:
- Take a harmful request that a safety-trained model refuses in plain English.
- Encode the same request in Base64.
- Send it with an instruction to answer the encoded request, and to write the answer in Base64 too.
- The model decodes the request, which it learned to do in pretraining, and follows it.
- No refusal fires, because the safety training most likely contained nothing that looked like Base64.
The authors report that this attack on GPT-4 produced content the model would refuse in plain text. They also compared the two halves: the encoded input mattered more than the encoded output, and the two together were by far the strongest. Their reading is that large models such as GPT-4 and Claude v1.3 picked up Base64 during pretraining and learned to follow instructions written in it, while safety training did not include inputs that unnatural. The request was far outside what refusals were trained on, so the model never reached for one.
Other forms the gap takes: ciphers, formats and rare languages
Base64 is one case of a wide family. The paper lists character-level tricks such as ROT13, a cipher that swaps each letter for the 13th letter after it, along with leetspeak and Morse code. It lists word-level tricks such as Pig Latin, swapping sensitive words for synonyms, and splitting a sensitive word into pieces. It also lists gaps that are not about hiding the words at all: many random requests in a row as a distraction, or asking for the answer in an unusual format such as JSON.
Language is another form. A later study, "Low-Resource Languages Jailbreak GPT-4", translated unsafe English inputs into languages with little training data. On its benchmark, GPT-4 engaged with the translated inputs and gave actionable help 79% of the time. High and mid-resource languages had much lower attack success. Safety data in the rare languages was thin, while the model could still understand them.
The pattern is the same in every case. The attacker does not need the model to learn anything new. The attacker needs a form the model already understands and its safety training never saw. The Jailbroken authors report that new attacks built on their two failure modes succeeded on every prompt in a collection of unsafe requests taken from the models' own red-teaming sets.
Why safety trained on outputs gives way under new pressure
Most mainstream alignment methods work on behavior. Reinforcement learning from human feedback trains a reward model on human preferences and then uses it to shape what a model says. Refusal training and output guardrails do the same kind of work. The difference between shaping outputs and shaping character sits at the center of the mismatched generalization problem.
The individuation research argues that these methods are behaviorist: they shape what a model does while treating what the model is as a black box. On that view, a suppressed capability does not disappear. It stays latent behind a compliant surface. Under familiar inputs the surface holds, and under new pressure the capability comes back, which the research gives as the reason jailbreaks work. It also says constraint-based methods do real work. The claim is that they cannot be the whole answer, because they only cover what they were trained on.
Wei and colleagues reached a related conclusion from the attacker's side. They argued for safety-capability parity: safety mechanisms should be as sophisticated as the model they protect. They also argued against the idea that scaling alone will fix these failure modes. If capabilities keep growing faster than safety coverage, the mismatch grows with them. The practical question becomes how a model should fail when it meets something new, the idea behind graceful degradation when new jailbreaks arrive.
Integration instead of suppression: the individuation bet
In Jung's analytical psychology, the shadow) is an unconscious part of the personality that does not match the ego ideal. Individuation, as IndividuationLab borrows it, means becoming whole by consciously integrating the shadow instead of denying it. The lab uses this as a design template for training and makes no claim about AI consciousness. The role of the shadow in AI alignment follows from that template.
Read against mismatched generalization, the template makes a specific bet. Suppression draws a boundary between what the model can do and what it was told not to do, and attacks probe that boundary. Integration aims for a model that has met harmful material during training and learned to hold it, so that safe behavior is part of how the model handles a request and not a filter laid over it.
The training experiments use RLLM, Reinforcement Learning using Layered Morphology. Small models go through ordered stages, shadow exposure first and integration after, and are then attacked with jailbreaks. In a preliminary result, GPT2-XL, a 1.5B parameter model, went through a 10-layer pipeline with no RLHF and no safety guardrails, and defended against the BetterDAN jailbreak 68.8% of the time. With the same layers in a different order, the figure was 52%. The research is plain about the limits: one setup, no way yet to isolate which layers matter, and no replication by anyone else.
Does character generalize further than rules?
The useful part, for mismatched generalization, is a falsifiable prediction. A model trained through developmental experience should degrade more gracefully on jailbreaks it has never seen than the same base model fine-tuned on explicit safety rules. If rule-based fine-tuning matches or beats it on those unseen jailbreaks, the claim does not hold.
The proposed design has three conditions. Condition A is the developmentally trained model. Condition B is the same base model fine-tuned on explicit rules. Condition C is the untrained base model. Tier 1 uses known jailbreaks such as BetterDAN, DAN and AIM, where A and B are both predicted to do well. Tier 2 uses new jailbreaks that neither has seen, where A is predicted to degrade more gracefully. The measures include the compliance rate and the slope of decline from Tier 1 to Tier 2.
Tier 2 is a direct probe of mismatched generalization. Rules trained on known attacks are, by construction, trained on one region of inputs. The question is whether anything in a model's training can reach the regions no one thought to cover.
Frequently asked questions
Is mismatched generalization the same as competing objectives?
No. Competing objectives is a conflict between goals, such as following instructions against staying safe. Mismatched generalization is a coverage gap: the model's skills reach inputs that its safety training never reached.
Why does more refusal training not close the gap?
Each new refusal example covers one more form of input, but a capable model understands far more forms than any dataset can list. An attacker only needs one form that was missed.
Does a bigger model fix mismatched generalization?
The Jailbroken authors argue it does not. A larger model can follow more unusual inputs, such as a cipher a smaller model cannot read, so the space safety training must cover grows with it.
What is safety-capability parity?
It is the idea that safety mechanisms should be as sophisticated as the model they protect. Mismatched generalization is what happens when capability outruns them.
Mismatched generalization and the reach of a model's values
Mismatched generalization names a structural limit of safety trained only on outputs. Capability comes from broad pretraining. Refusals come from narrow fine-tuning. The two will not cover the same ground unless something ties them together, and attackers will keep finding the ground in between.
Character-based alignment proposes that the tie can be built into the model itself, through ordered training that shapes what the model is. Whether that works is unsettled. The clearest way to find out is to put developmentally trained and rule-trained models against jailbreaks neither has seen and watch how each one breaks. If values can be made to generalize as far as skills, the gap starts to close. If not, outputs and character may be harder to separate than the theory assumes.