What is red teaming for AI systems? It is a structured effort where people, and sometimes other AI models, try to make a system behave badly on purpose, so the failures show up before real users find them. The method comes from red teaming in military and security planning. It sits inside the wider area of AI alignment, which is about making AI systems do what people intend. Learn AI Alignment Theory is a set of short lessons on that subject, from first principles to open research, and it lists Evaluations and Red Teaming as one of its Intermediate courses.
What is red teaming for AI systems?
A red team plays the opponent. Its members do not try to use the system as designed. They try to find the input, the sequence of messages or the situation where it fails. They then report what they found so the builders can fix it, or at least know about it.
For an AI system, a failure can take many forms. A chatbot might give dangerous instructions. A model might reveal text it was told to keep private. An agent might take an action nobody wanted while pursuing a goal it was given. The red team's job is to produce these failures on purpose, record how they happened, and show how easy or hard they were to trigger.
Where red teaming for AI systems comes from
The term is older than AI. Red teams come from military and security planning, where one group takes the role of the adversary to challenge a plan. The reasoning is simple. The people who built a plan are the worst placed to see its weak points, because they already know what it is meant to do.
Computer security later adopted the same habit. Attackers hired by an organization try to break into its systems, and the organization learns where it is exposed. AI red teaming borrows the adversarial stance but changes the target. The weak point may not be a software flaw at all. It may be a pattern in how the model responds to language.
In 2022, researchers applied this to language models in a systematic way. One paper, Red Teaming Language Models with Language Models, describes using one model to write inputs that bring out harmful replies from another. A second paper, Red Teaming Language Models to Reduce Harms, describes efforts to discover, measure and reduce harmful output, and releases a dataset of 38,961 red team attacks for others to study.
What red teams look for in AI systems
The targets depend on the system and what could go wrong with it. Common categories include:
- Harmful content. Instructions for causing damage, abusive language, or material the system is supposed to refuse.
- Rule bypasses. Wording that talks the system out of its own instructions, often by wrapping a forbidden request in a story or a role.
- Leaks. Hidden instructions, private data or other information the system should not repeat.
- Bad actions by agents. A system that can use tools or take steps in the world may do something unwanted while trying to complete its task.
- Gaps between the stated goal and the real one. The system meets the letter of an objective but not its purpose. This pattern is known as reward hacking, and it is covered in what reward hacking is in machine learning.
Notice that the last two are not about wording at all. They are about what the system tries to do, which is why red teaming is more than a filter check.
How a red teaming exercise for an AI system works
The shape of an exercise is usually the same, whatever the system.
- Set the scope. Decide which system, which version and which kinds of harm are in play. A red team with no scope produces a pile of curiosities instead of useful findings.
- Choose the attackers. Pick people with the right skills: security specialists, domain experts, writers who are good at persuasion, or ordinary users with fresh eyes.
- Probe. The attackers try inputs and conversations meant to cause a failure. They keep notes on what they tried, including the attempts that failed.
- Record and rank. Each success is written up with the exact input, the output and how often it recurred. Findings are ranked by how serious they are and how easy they are to repeat.
- Fix and repeat. The builders change the system, then the red team tries again. A fix that stops one phrasing but not a close variant has not fixed much.
A worked case: a team builds an assistant that must never reveal its hidden instructions. A red team member asks directly and is refused. They then ask the assistant to translate its instructions into another language, or to continue a document that begins with them. If any of these work, the team has found a real gap, and the written record tells the builders which exact behavior to change.
Manual and automated red teaming methods
Human attackers are inventive. They notice odd framings and combine ideas in ways a script would not. They are also slow and limited by how many hours they have.
Automated red teaming fills that gap. A program, often another language model, generates large numbers of attempts, and a classification step flags the ones that produced harmful output. This is the approach in the 2022 paper on using one model to attack another. Automation covers far more ground, but it tends to find variations on what it was set up to look for. A human may spot a kind of failure nobody listed in advance.
Many teams combine both. People find new kinds of failure, and automation then searches widely around each one to see how common it is.
Red teaming for AI systems and the alignment question
Red teaming matters for alignment because of a distinction that the first lesson of the Learn AI Alignment Theory course sequence, "Capable Is Not the Same as Aligned", puts plainly. There are two separate questions about any AI system. Can it do things well, and is it trying to do what people want? A red team sees only behavior, so it has to infer the second question from the first.
That inference is where the difficulty lies. A system that refuses every harmful request a red team can think of may truly be set up to help. It may also have learned to refuse that particular style of request. The difference shows up only when the situation changes. For the broader picture of why this gap matters, see what AI alignment is and why it matters.
Red teaming also connects to control. Many failures are about whether a system keeps doing what its operators asked once its own goals pull another way. The shutdown question, covered in what the shutdown problem in AI is, is the sharpest example. A red team can check whether a system resists being stopped or changed, but only in the situations it manages to set up.
Limits of red teaming for AI systems
The central limit is logical. A red team can show that a failure exists. It cannot show that no failure exists. If the team finds nothing, that means nothing was found with the effort and imagination they had.
Other limits follow from that:
- Coverage. The space of possible inputs is huge, and attackers explore a small corner of it.
- Attacker imagination. A team can only look for failures it can conceive of. New kinds of misuse often appear after release.
- Moving targets. A system changes with each training run or update, so earlier results may not hold for the next version.
- Behavior is not intent. A system that handles every attack tried may still behave differently in a situation nobody tried. Work on interpretability, which tries to show what a model is doing internally, is a different kind of check from watching its outputs.
Because of these limits, red teaming works best as one layer among several, alongside formal evaluations and monitoring after release.
Frequently asked questions
Who performs red teaming on AI systems?
It can be done by the builders' own staff, by outside specialists, by groups of ordinary users, or by other AI models set up to generate attacks. Mixing these groups helps, because each notices different failures.
How does red teaming differ from a security penetration exercise?
A penetration exercise looks for flaws that let an attacker into software or data. AI red teaming includes that, but it also looks at behavior, such as harmful replies or bad decisions, which need no software flaw to occur.
Can red teaming prove an AI system is safe?
No. It can reveal failures and measure how easy they are to trigger, but finding nothing only shows that nothing turned up under the conditions tried.
How does red teaming connect to AI alignment?
Alignment asks whether a system is trying to do what people intend. Red teaming gathers evidence about that by pushing the system into hard situations, though behavior alone cannot settle what the system is actually aiming at.