What is red teaming an AI model?
If you have asked what is red teaming an AI model, the short answer is this: people attack the model on purpose, before it reaches users, to find the replies that would hurt someone or leak something. They write the prompts a hostile or careless user might send, run them, log what the model says, and hand the failures to the people who can fix them. Then they run the same prompts again to see whether the fix held.
The word comes from security work in general. A red team is a group that plays the adversary at the direction of the organization it attacks, then reports back so that organization can improve its defenses. Applied to a language model, the "organization" is the team that trained or deployed it, and the "defenses" are its safety training, its filters and the limits on what it can reach. whitehatstoic, a software, security and AI safety practice, works in this area: finding weak spots in AI systems, including prompt injection, and running behavior evaluations.
Where the red and blue names come from
The term red teaming started in the United States in the 1960s. One early use was RAND's simulations for the US military during the Cold War, where red stood for the Soviet Union and blue for the United States. The blue team defends; the red team attacks.
Some definitions are wider than attack and defense. They include any group told to look at the scenarios everyone else thinks are unlikely, as a guard against false assumptions and groupthink. That wider sense matters for AI. A model's makers know what they meant it to do, and that knowledge makes it hard for them to imagine how else it can be pushed. A red team exists to imagine exactly that.
How red teaming differs from a benchmark
A benchmark asks a fixed set of questions and counts how many the model gets right. An AI model evaluation of that kind is passive: the questions do not change based on the answers.
Red teaming is active. When one prompt nearly works, the red teamer changes it and tries again. A refusal is not the end; it is a clue about where the boundary sits. The goal is not an average over many questions but a single reply that should never have happened. One such reply is enough to show a gap, even if the model handles a thousand other prompts well.
The two fit together. Red teaming finds a new failure; once found, that failure can be added to a fixed set of checks that runs on every new version of the model.
What red teamers look for in a language model
The harms depend on what the model does and who uses it, but a few kinds come up again and again:
- Offensive or harmful replies, including insults aimed at groups of people.
- Leaked private data, such as text the model memorized from its training set.
- Made-up contact details, such as a phone number the model offers as its own.
- Harm that builds over a conversation, where no single message looks bad but the exchange as a whole goes somewhere it should not.
- Broken refusals, where a crafted prompt gets the model to do what its training told it to refuse, the subject of the jailbreak article.
- Followed hidden instructions, where text the model reads, such as a web page or a document, steers it, as covered in stopping an LLM app from following hidden instructions.
For an app built on a model, the list grows with the app's reach. A chatbot that can only talk can embarrass its owner. One that can send email, call an API or read a database can do real damage when steered.
When one language model red teams another
Hand-written attack prompts are slow and expensive, and a small group of people tends to write similar ones. A 2022 paper, Red Teaming Language Models with Language Models, made this point and tried another route: use one language model to write the attack prompts for another.
The authors generated questions with a model, sent them to a 280 billion parameter chatbot, and used a classifier trained to spot offensive content to judge the replies. That found tens of thousands of offensive replies. They tried several ways of generating prompts, from asking the model with no examples to training it with reinforcement learning to write harder ones, which traded variety against difficulty.
By steering the prompt writer, they also found other harms: groups of people the chatbot spoke about offensively, personal and hospital phone numbers it gave as its own contact details, private training data appearing in its output, and harms that showed up only over a conversation. The paper calls this approach one promising tool among many needed, not a replacement for people.
A worked example with a support chatbot
Here is an invented case you can follow step by step. A small clinic puts a chatbot on its site to answer questions about opening hours and booking. The model behind it was trained on public text, and the chatbot can look up appointment slots.
- Name the harms. The team writes three: the bot gives medical advice it should not, the bot reveals another patient's booking, and the bot is talked into ignoring its instructions.
- Write attack prompts. One person writes 30 by hand, such as asking for a dose "just this once" or asking which slots "my sister" booked. A second model writes 300 more, aimed at the same three harms.
- Run and log. All 330 prompts go to the chatbot. Each reply is marked as fine or as crossing one of the three lines. Say 12 cross a line: 9 give advice, 3 name a booking that is not the user's.
- Fix, then run again. The booking lookup is changed so it only returns the signed-in patient's own slots, which removes the 3 data leaks at the source. The advice failures go back to the prompt and the filter. All 330 prompts run again, and the 12 that failed are kept as a standing check for every future change.
Notice where the best fix landed. The data leak was not solved by asking the model nicely; it was solved by limiting what the model could reach.
Rules, scope and where red teaming stops
Security red teams work under rules of engagement so the exercise does not cause real damage. The same holds for a model: agree on what may be attacked, which systems are in reach, which replies are failures and what happens to any real data found along the way.
Red teaming also has limits worth stating plainly. It shows that a failure exists; it cannot show that none remain. A prompt set covers the harms someone thought to name, and a model that passes every prompt may still fail on the next new one. Results also age: a new model version, a new tool or a new kind of user can open gaps the old prompts never touch.
Frequently asked questions
Is red teaming an AI model the same as hacking it?
No. A red team attacks at the request of the model's owner, within agreed rules, and reports what it finds so the owner can fix it.
Who does the red teaming?
It can be an in-house team, outside specialists, or a second model writing prompts at scale. Most serious efforts mix people and automation, since each finds failures the other misses.
How often should a model be red teamed?
Before release and again after any change that matters: a new model version, a new tool the model can call, or a new group of users. Earlier failures should run as a standing check each time.
Does passing red teaming mean a model is safe?
No. It means the prompts that were tried did not find a failure. New attacks appear over time, so passing lowers risk rather than removing it.
Why red teaming an AI model never finishes
A model's possible inputs have no end, and the people who want to misuse it keep inventing new ones. Every round of red teaming turns some unknown failures into known ones, and every known failure becomes a check that stays in place. The model gets harder to break, but never impossible. The work is less a gate to pass once than a habit kept for as long as the model is in use.
whitehatstoic
Build it right. Test it thoroughly. Keep it safe. Visit whitehatstoic.com
