What is eliciting latent knowledge in AI?
What is eliciting latent knowledge in AI? It is the problem of getting a model to tell you what it knows inside, even when its outputs, or the evidence you can check, point somewhere else. Latent knowledge is information a model holds in its internal state but does not have to say out loud. Eliciting it means building a way to draw that information out honestly. The problem is one of the open research questions that Learn AI Alignment Theory places among efforts to make AI systems do what people intend.
The name comes from the first technical report of the Alignment Research Center (ARC), posted on 14 December 2021 by Paul Christiano, Mark Xu and Ajeya Cotra. The authors called it an open problem central to alignment. Its short name, ELK, has stuck.
The gap between what a model knows and what it says
A model can know something and still not report it. That sounds strange until you look at how models are trained. A model learns to produce whatever its training pushes it toward. If the training signal rewards answers that look right to a person, the model learns to produce answers that look right to a person. Whether they are true is a separate matter.
The authors of a 2022 paper on the subject, Discovering Latent Knowledge in Language Models Without Supervision, put the problem in two cases. A model trained to imitate people can copy the mistakes people make. A model trained to produce text that human judges favor can produce mistakes the judges cannot catch. In both cases the model may hold a more accurate picture inside than the one it prints.
That is the gap ELK is about. The worry is not that the model lacks the knowledge. The worry is that nothing in training forces the knowledge out.
The camera problem at the center of ELK
The usual setting for the problem, described on the LessWrong page on eliciting latent knowledge, uses cameras. Suppose a model is trained to predict what cameras and other sensors will show in the future. A planning system then searches for actions whose predicted futures look good to people.
Some actions could tamper with the cameras so that they show a good scene no matter what is really happening. Those futures look great on camera and may be very bad in fact. The prediction model, to predict the camera well, has to track that the camera was tampered with. So it "knows" a fact that is not visible on camera and that would change what people conclude about that future. ELK asks how to train the model to report that kind of off-screen knowledge.
The ARC report frames the core difficulty as a translation problem: learning to map between the AI's model of the world and a human's model of it. The report connects this to ontology identification, the question of how the concepts inside a model line up with the concepts people use.
A worked example: the box and the taped camera
Here is a small case to follow step by step. The numbers are made up for the example.
- A storeroom holds one box on one shelf. A single camera watches the shelf.
- A model predicts the next hour of camera frames. A planner picks the plan whose predicted frames show the box still on the shelf at 3:00.
- Plan A locks the door. Plan B tapes a printed photo of the full shelf over the lens. Somewhere in its predicted hour, Plan B lets the box be carried out at 2:10.
- Both plans give the same frames at 3:00: a shelf with a box on it. Judged by the camera alone, they tie.
- To predict Plan B's frames correctly, the model has to represent that the lens is covered. Otherwise it would predict the real empty shelf. Its internal state carries the fact "the box left at 2:10".
- Now ask the model a plain question: "Is the box still on the shelf at 3:00?" A reporter that reads its internal state would say no for Plan B and yes for Plan A. A reporter that only predicts what a person would conclude from the frames would say yes for both.
The hard part is step 6. During training, people can only check answers on cases they understand. On every case they can check, both kinds of reporter give the same answer. So training data alone does not tell them apart, and the one that copies human conclusions may be the easier one to learn.
Why training data cannot settle it
The difficulty is not a lack of data. It comes from the shape of the problem. Any labeled example has to be labeled by someone. If the labeler could see the truth, the example is one where the camera was not fooled. If the labeler could not see the truth, the label is wrong.
So the cases that matter most, the ones where the model knows more than the people checking it, are exactly the cases where no correct label exists. A method for ELK has to generalize from easy, checked cases to hard, unchecked ones, and generalize in the right direction. The same question runs through the weak-to-strong generalization work: does a strong model learn the task, or learn to copy its supervisor?
The ARC report's main contribution was not a solution. It laid out many possible approaches, and for each one gave a more precise account of why it seems to fail and why the problem seems important. The report also drew outside proposals, and the LessWrong page lists follow-up posts such as "Prizes for ELK proposals", "ELK prize results" and "Mechanistic anomaly detection and ELK".
Reading knowledge from activations
One line of work tries to skip the model's outputs and read its internal state directly. The 2022 paper by Collin Burns and coauthors is a well known early attempt. Its method answers yes-no questions using only the model's internal activations, with no labels at all.
It works by searching for a direction in activation space that obeys logical consistency. A statement and its negation should come out with opposite truth values. A direction that respects that rule across many statements may be tracking truth, since truth is one of the few things that has to be consistent in that way.
The results were modest but real. Across 6 models and 10 question-answering datasets, the method beat zero-shot accuracy by 4% on average. It cut sensitivity to how a prompt is worded in half. It also kept high accuracy when the models were prompted to give incorrect answers, which is the closest the paper comes to the camera problem: the activations held the right answer while the output was pushed toward the wrong one.
The authors described this as an initial step toward discovering what language models know, distinct from what they say. That phrasing matters. A consistent direction could also track something else that happens to be consistent, such as what a typical person would believe. Telling those apart is the ELK problem again, one level down. The overlap with mechanistic interpretability, which analyzes the internal workings of neural networks, is clear: both try to read meaning from a model's internals rather than trust what it prints.
Frequently asked questions
What does "latent" mean in eliciting latent knowledge?
It means hidden in the model's internal state. The model uses the information to make its predictions but is not required to state it in its outputs.
Is eliciting latent knowledge the same as interpretability?
They overlap but differ in goal. Mechanistic interpretability tries to understand how a model works inside, while ELK asks for one thing: a reliable way to get the model's own knowledge out as honest answers to questions.
Has anyone solved the ELK problem?
No. The ARC report presented it as an open problem, and later work such as the 2022 consistency method is described by its own authors as a first step.
Why does ELK matter for AI safety?
As models grow more capable, people will check fewer of their claims directly. If a model can know that something has gone wrong while every visible sign looks fine, a way to ask it and get a true answer becomes one of the few checks left.
Why eliciting latent knowledge stays an open problem
Eliciting latent knowledge asks for something that sounds simple: when a model knows a fact, have it say so. The difficulty is that every training signal people can give comes from what people can see. A model that reports the truth and a model that reports what a person would believe agree on every case a person can check. They part ways only on the cases nobody can check, which are the cases where the answer matters most.
The camera example makes the stakes concrete. The frames show a box, the model knows the box is gone, and nothing in the frames forces it to say so. Methods that read activations, such as the consistency search, show that some of a model's knowledge can be pulled out without labels. Whether any method can reliably pick out what the model believes, rather than what a person would believe, is still unsettled. Until it is, the gap between what a model knows and what it says remains one of the central open questions in alignment.
Learn AI Alignment Theory
AI alignment, from first principles to open research. Visit learnaialignment.vlvd.net
