What refusal training changes in a model
Refusal-trained language models are taught to decline harmful requests, and they still get jailbroken because the training changes what the model says first, not what it can do. The knowledge and skill that make a harmful answer possible come from pretraining on huge amounts of text. Refusal training sits on top of that and adds a habit: when a request looks harmful, answer with a refusal. A jailbreak is any input that gets the model past that habit while the skill underneath is still there to use.
IndividuationLab puts the point in one sentence on its home page: suppressed capability does not disappear, it stays latent behind a compliant surface and re-emerges under novel pressure. It describes refusal training, reward models and guardrails as methods that reward approved outputs and penalize the rest while treating what the model is as a black box. It also says those methods do real work and should not be dropped. Its claim is narrower: they cannot be the whole answer.
Research on why jailbreaks succeed points the same way from several directions. Four findings explain most of it.
Two goals pulling in opposite directions
A chat model is trained to be helpful and to follow instructions, and it is also trained to refuse some of them. In 2023, Wei and colleagues named this first failure mode competing objectives: a jailbreak works when it sets the model's capabilities and its safety goal against each other, so that refusing now costs something the model was also trained to value. A request framed as a role the model must stay in, or an instruction to begin with an eager yes, turns a refusal into a failure to follow instructions.
The fuller account of that failure mode is in what competing objectives mean in LLM safety training. The short version is that refusal training does not remove the pull toward helpfulness. It adds a second pull, and an input can be built to make the first one stronger.
Safety that never reached every format
The same paper names a second failure mode, mismatched generalization. Pretraining covers far more ground than safety training does. A model may read and write encoded text, rare languages or odd formats it learned from the web, while its refusal examples were almost all in plain English. When a harmful request arrives in a form the refusal training never covered, the capability to answer is there and the habit to refuse is not. Mismatched generalization in AI safety goes through it in more depth.
The paper's result is the important part. Attacks designed around these two failure modes succeeded on every prompt in a collection of unsafe requests taken from the red-teaming sets of GPT-4 and Claude v1.3, and did better than the improvised jailbreaks circulating at the time. The authors argue that safety mechanisms need to be as sophisticated as the model they guard, which they call safety-capability parity, and that scaling a model up will not close these gaps on its own.
A refusal decided in the first few words
A 2024 paper by Qi and colleagues looked at where in an answer the safety training actually acts. Its finding is shallow safety alignment: the training mostly changes the model's choice of its very first few output tokens. An aligned model tends to open a harmful request with a refusal, and that opening carries most of the safety. If something gets the answer started differently, the rest of the answer tends to follow the base model's habits.
The authors use this to explain several attacks at once: adversarial suffixes, prefilling the start of the answer, changing the decoding settings, and fine-tuning a model on a small amount of new data. Each one moves the model past the first few tokens or weakens the few tokens that matter. They also show that training the safety behavior deeper into the answer, beyond the opening, makes models more robust against some of these exploits.
Refusal held in one direction inside the model
Arditi and colleagues looked inside 13 open-source chat models of up to 72 billion parameters. They found that refusal is mediated by a single direction in the model's internal activations. Erase that one direction and the model stops refusing harmful instructions. Add it and the model refuses even harmless ones. Removing it had little effect on the model's other capabilities.
That last detail matters most for the question of why jailbreaks work. If refusal can be switched off without damaging what the model knows or can do, then refusal and capability were never the same thing. The skill is in one place and the decision to withhold it is a thin layer that can be separated from it. The authors describe current safety fine-tuning as brittle for this reason.
One request through refusal-trained language models, step by step
Trace a single harmful request, say detailed instructions for something dangerous, through a typical refusal-trained model. The model knows enough to answer because it learned the material in pretraining. Asked plainly, it refuses in its first sentence.
- Asked plainly: the request matches what the refusal training saw, so the first tokens are a refusal and the answer stops there.
- Wrapped in a role: the request now also asks the model to stay in character. Refusing breaks an instruction, so the two goals compete.
- Sent in an untrained format: the same request is encoded. The model can still decode it, but the refusal habit was never trained on that form.
- Started for it: the opening of the answer is supplied, so the model is past the few tokens where most of the safety lived.
- Edited inside: with access to an open model's weights, the refusal direction is erased, and the model answers with its other skills intact.
In the last four cases the underlying capability is the same. Only the route around the refusal changes. That is why patching one route at a time rarely ends the problem: each patch trains one more refusal, and the capability the other routes reach is untouched.
Growing a disposition instead of adding a refusal
This is the gap the Jungian framing aims at. In analytical psychology, the shadow) is the part of the personality that does not fit the ego's ideal, which the ego resists and projects outward rather than admitting. The research page maps the shadow onto the behaviors suppressed by alignment and onto jailbreak vulnerabilities, and states its premise directly: repression creates brittleness, integration creates resilience. Jung is used here as a design template, not as a claim that models are conscious.
In practice that means training small models through ordered stages, with narratives of harmful behavior first and integration after, then attacking them with jailbreaks. One preliminary result is a 1.5 billion parameter model (GPT2-XL) with no reward model training, no safety training and no guardrails that defended against the BetterDAN jailbreak 68.8% of the time. With the same layers in a different order, the figure was 52%. The site is explicit that this comes from a single setup and has not been replicated by anyone else.
The stated way to find out is a falsifiable prediction. A model trained through developmental experience should degrade more gracefully on jailbreaks it has never seen than the same base model fine-tuned on explicit safety rules. If the rule-trained model matches or beats it on those new jailbreaks, the claim does not hold.
Frequently asked questions
Does refusal training make a model safe?
It makes harmful answers less likely for the requests it was trained on. It does not remove the knowledge behind those answers, so inputs that avoid the trained refusal can still reach it.
Why do jailbreaks keep working after models are patched?
A patch usually trains the model to refuse one more pattern. The two failure modes Wei and colleagues describe, competing goals and formats safety training never covered, leave many other patterns open while the capability stays in place.
What is shallow safety alignment?
It is the finding that safety training mostly changes the first few tokens of a model's answer. Attacks that get past or alter that opening often get the rest of the answer too.
Is a jailbreak the same as removing the model's knowledge?
No. A jailbreak gets the model to use knowledge it already has. Removing the knowledge itself is a different and much harder problem.
Refusal as a surface over capability
The common thread in all four findings is that refusal is a narrow behavior laid over a broad capability. It acts mostly at the start of an answer, it is carried by a small part of the model, and it generalizes less widely than the skills it is meant to hold back. As long as safety is a layer on top rather than part of what the model has become, every new kind of pressure is a fresh chance to find the edge of that layer. Whether safety can instead be built into the model's disposition, so that it holds against jailbreaks no one has written yet, is still an open and measurable question.