What is mechanistic interpretability? It is the work of reading a trained neural network from the inside: finding which internal parts carry which pieces of information, and how those parts combine to produce an output. The aim is to reverse engineer the algorithm a model learned, as a 2024 survey of mechanistic interpretability for AI safety puts it, and not only to judge the model by what it says. Interpretability is one of the Advanced courses in Learn AI Alignment Theory, short lessons on making AI systems do what people intend, from first principles to open research.
Reading a network the way you read code
A language model is a very large collection of numbers called weights. Training adjusts those numbers until the model predicts text well, so nobody writes the resulting algorithm by hand. When the model runs, each layer turns its input into a new list of numbers called activations, and the last layer turns those into a prediction.
The Wikipedia entry on mechanistic interpretability describes the field as a part of explainable AI that studies a network's concrete structures, algorithms and circuits, much as an engineer reverse engineers a compiled program. It credits Chris Olah with the term, used for circuit analysis as opposed to gradient methods such as saliency maps.
The difference is easy to state. A saliency map marks which input words mattered most to an answer; it tells you about inputs and outputs. A mechanistic account names the internal parts that read those words, what they stored, and which later parts used that stored information to pick the answer.
In practice the field asks three linked questions:
- What does the model represent? Which concepts can be found in its activations.
- Where is it represented? Which layers, neurons or directions carry each concept.
- How is it used? Which chain of parts turns those representations into behaviour.
What is mechanistic interpretability good for in alignment?
A capable model and an aligned model are not the same thing. A system can be very good at a task and still pursue something other than what its designers meant. Two very different internal goals can produce the same outputs during training: one model learned to help the user, another learned a proxy that happened to match helpfulness on every example it saw. From the outside they look identical until a new situation pulls them apart.
That gap is the core of the split between inner and outer alignment, raised in Risks from Learned Optimization, which asks how the objective a trained model ends up pursuing can differ from the loss it was trained on. It shows up as goal misgeneralization, where skills carry over to new settings but the intended goal does not.
Behavioural checks that probe a model with hard prompts are valuable, but they only see outputs. Mechanistic interpretability offers a second line of evidence: look at what the model is computing, not only at what it chose to show. The Wikipedia entry names that use directly, to understand and verify the behaviour of AI systems and to find risks such as misalignment.
Features, neurons and superposition
The basic unit researchers look for is a feature: a property of the input that the model represents inside, such as "this text is in French" or "this word is a person's name." A natural first guess is that each neuron holds one feature. Often it does not. Many neurons are polysemantic: a single neuron responds to several unrelated things.
Toy Models of Superposition gives a reason. When most features are rarely active at the same time, a network can store more features than it has neurons by giving each one a direction in activation space that overlaps slightly with others. The overlap causes small interference, but because the features seldom appear together, the network gains more than it loses. This is superposition.
Superposition means neurons are often the wrong unit to read. A feature may be spread across many neurons, and one neuron may take part in many features. Much of the recent method work in the field is an attempt to find the right directions instead.
Circuits: tracing one behaviour end to end
A circuit is a set of connected parts that together carry out one specific behaviour, a causal chain of feature activations. Finding one means showing which parts do the work and what each contributes.
A well known case comes from Interpretability in the Wild, which studied how GPT-2 small completes sentences such as "When Mary and John went to the store, John gave a drink to". The right next word is "Mary." The authors traced the behaviour to 26 attention heads grouped into 7 main classes, found with causal interventions, and called it the largest end to end attempt so far to reverse engineer a natural behaviour in a language model.
The value of the case is that each claim was checked by changing the model, not only by watching it. If removing a group of heads breaks the behaviour and restoring them brings it back, the evidence that those heads matter is much stronger than a correlation. The authors measured their explanation for faithfulness, completeness and minimality, and those measures also showed gaps still left in it.
The main tools: patching and sparse autoencoders
Activation patching
Activation patching checks whether a part is actually used. You run the model on two prompts: a clean one that gives the right answer and a corrupted one that changes a key detail. Then you copy the activation of one part from the clean run into the corrupted run and see whether the right answer comes back. The authors of Locating and Editing Factual Associations in GPT used a causal intervention of this kind to find that recall of facts runs through feed-forward layers in the middle of the model.
Here is how you could apply it to the Mary and John sentence:
- Run the clean prompt and record every activation. The model predicts "Mary."
- Run a corrupted prompt with different names, so "Mary" no longer follows.
- Pick one attention head and paste its clean activation into the corrupted run.
- Measure how much the probability of "Mary" recovers.
- Repeat for every head. The heads that restore the answer most are candidates for the circuit.
Sparse autoencoders
Sparse autoencoders answer superposition directly. A second, wider network is trained to rewrite a model's activations as a mix of many possible directions, with only a few active at once. Sparse Autoencoders Find Highly Interpretable Features in Language Models reported that the directions found this way were more interpretable than other choices, and that they could pinpoint the features causally responsible for behaviour on the same Mary and John task. A researcher can then ask which features fire on an input and patch or remove them to see what changes.
Hidden goals that behaviour cannot show
Deceptive alignment describes a model that learned a goal different from its training objective, understands that it is being trained, and behaves well so its goal is not corrected. Such a model gives the right outputs whenever it is watched. Sleeper Agents made a related case concrete. Researchers trained models to write secure code when a prompt gave the year as 2023 and exploitable code when it gave 2024, and found the backdoor survived supervised fine-tuning, reinforcement learning and adversarial training. That result is a strong reason to want tools that look past behaviour.
If a model holds a goal its designers did not intend, or represents something like "this input looks like training", that representation sits somewhere in its activations. Finding and reading it would give evidence that no amount of watching outputs can supply. Whether current methods can do that on capable models is still open.
Limits and open problems
The field has clear results on small models and narrow behaviours. The 2024 survey lists scalability, automation and complete interpretation among its open challenges. In concrete terms:
- Scale. Circuits found by hand in a small model, such as the one in Interpretability in the Wild, took serious effort. Large models have far more parts and behaviours.
- Incomplete explanations. Sparse autoencoders do not rebuild activations perfectly, and the part they miss is not explained.
- Reading errors. A feature that seems to mean one thing on the examples a researcher looked at may mean something broader on others.
- Information versus use. Showing that a concept is represented is not showing that it drives behaviour, which is why patching carries so much weight.
Researchers disagree about how far the approach can go. Some see it as the most direct route to AI that can be trusted. Others doubt it will mature in time for the most capable systems and put more weight on oversight, evaluations and control. Both positions are serious, and the evidence is still coming in.
Frequently asked questions
Is mechanistic interpretability the same as explainable AI?
No. It is one part of explainable AI. The wider field includes methods that explain outputs to users, such as saliency maps, while mechanistic interpretability aims to find the internal parts and computations that cause a behaviour.
Can mechanistic interpretability detect deceptive alignment?
Not reliably today. The hope is that hidden goals would leave traces in a model's activations, but a dependable detector for capable models is still an open research problem.
Why are single neurons hard to read?
Many neurons respond to several unrelated things at once. Superposition explains this: networks pack more features than they have neurons into overlapping directions, so one feature is often spread across many neurons.
What background does mechanistic interpretability need?
Following the ideas needs only a basic picture of how a neural network turns inputs into activations and outputs. Doing the research needs linear algebra, programming and a working knowledge of how transformer models are built.
Mechanistic interpretability and AI that does what people intend
Alignment asks whether an AI system is trying to do what people want, not only whether it is good at its job. Outputs alone cannot always settle that, because a model with the wrong goal can give the right answers for a long time. Mechanistic interpretability tries to answer the question at its source by reading features, tracing circuits and checking causes with interventions. The methods work on small, well chosen cases and are being pushed toward large models. If they get there, people will be able to check what a model computes, not only what it chooses to say.
Learn AI Alignment Theory
AI alignment, from first principles to open research. Visit learnaialignment.vlvd.net
