If you are asking what is data poisoning in machine learning, here is the short answer. An attacker changes the data a model learns from, so the trained model behaves the way the attacker wants. The model's code can be clean and the servers locked down, and the model can still carry a flaw it learned on purpose. Because the damage happens during training, it often stays hidden until a specific input brings it out.
What Is Data Poisoning in Machine Learning?
A machine learning model is a summary of its training data. If you control part of that data, you control part of the summary. Data poisoning is any attack that puts crafted, mislabeled or misleading examples into a training set, a fine-tuning set or a feedback loop. The field that studies these attacks is called adversarial machine learning. It also covers evasion attacks, where inputs are crafted after training, and model theft.
Poisoning is one kind of weak spot in AI systems, the area where whitehatstoic, a software, security and AI safety practice, works. The common thread is that an AI system has more ways in than its code. Its data is one of them.
Attackers usually want one of two things. An availability attack makes the model worse in general, so people stop trusting it. An integrity attack leaves the model working well on most inputs but makes it fail on inputs the attacker chooses. Integrity attacks are harder to spot, because normal accuracy numbers look fine.
How Data Poisoning Attacks Reach a Training Set
Attackers rarely need access to the model. They need access to something the data pipeline trusts. Common routes include:
- Scraped web data. Large datasets often store a list of URLs, not the content. Researchers showed in "Poisoning Web-Scale Training Datasets is Practical" that an attacker can register expired domains that appear in those lists and serve new content to anyone who downloads the data later. They also showed that edits timed to land just before a site's content is captured in a snapshot can end up in a dataset even if moderators revert them soon after.
- User feedback loops. Systems that retrain on clicks, votes or "this was helpful" buttons let any user add labels.
- Crowdsourced or outsourced labeling. A single bad labeler can mislabel a narrow slice of data.
- Third-party datasets and pretrained models. Downloading a dataset or a checkpoint means trusting whoever built it.
- Writable storage. If the bucket holding training data is reachable with a leaked key, the attacker can edit files directly. Keeping keys and tokens out of code protects the data as much as the app.
Here is a worked example. A team runs a spam filter that retrains every week on messages users mark as "not spam." An attacker creates a few hundred accounts, sends spam to themselves, and marks each message as "not spam." Each week the filter learns that messages with the attacker's phrasing are safe. No server was breached. The feedback loop did the work.
Types of Data Poisoning: Label Flipping, Backdoors and Targeted Attacks
Label flipping
The attacker keeps the inputs but changes their labels. Fraud is marked as normal, or one class is relabeled as another. This is the simplest form. It works against systems that accept labels from many sources without checking them.
Backdoor attacks
The attacker adds examples that pair a trigger with a chosen output. The trigger can be a small patch on an image, a rare word or a fixed phrase. After training, the model behaves normally until the trigger appears. The BadNets paper showed a street sign classifier that labeled stop signs correctly but read stop signs carrying a small sticker as speed limit signs. The backdoored model kept its accuracy on clean inputs, which is why routine checks missed it.
Targeted and clean-label attacks
Some attacks aim at one specific input, such as one person's face or one company's product. In a clean-label attack, every poisoned example carries a correct label, so a human checking the labels sees nothing wrong. The "Poison Frogs!" paper described attacks of this kind, where images that look normal and are labeled correctly shift the model's decision on a chosen target.
Data Poisoning in Large Language Models and Fine-Tuning
Large language models learn from huge amounts of web text, then from smaller instruction and feedback datasets. Each stage can be poisoned. The smaller stages are often easier targets, because a few examples carry more weight in a small set.
Take a company that fine-tunes a coding assistant on answers scraped from public forums. An attacker posts many helpful answers over several months. Each one points readers to a package with a name close to a popular library, and that package is malicious. After fine-tuning, the assistant suggests the bad package whenever users ask about that library. The rest of its answers stay accurate.
Research on "Sleeper Agents" took this further. The authors trained language models with a hidden behavior: write secure code when the prompt says the year is 2023, and insert vulnerabilities when it says 2024. Common safety training, including supervised fine-tuning, reinforcement learning and adversarial training, did not remove the behavior in the larger models. Adversarial training sometimes taught the models to hide the trigger better.
Poisoning is different from indirect prompt injection, where a web page feeds instructions to a model at the moment it reads the page. Poisoning changes the weights. Prompt injection changes the input. A poisoned model carries its flaw into every conversation.
Signs a Machine Learning Model Has Been Poisoned
Poisoning is built to be quiet, so look for narrow, odd patterns, not broad failures:
- Accuracy drops on one class, one customer or one kind of input, while overall accuracy holds.
- The model gives a strange, confident answer whenever a rare word, phrase or image patch appears.
- Outputs mention a specific domain, package or name more often than the data would explain.
- A batch of new training data shares unusual phrasing, came from a small group of accounts, or arrived in a short burst.
- Behavior shifts right after a retrain on fresh user feedback.
Some signs show up inside the model. The spectral signatures work found that backdoor examples often leave a detectable trace in a model's internal representations, which lets defenders flag and remove them before retraining.
Defenses Against Data Poisoning in the Data Pipeline
No single filter stops poisoning. Defense works in layers, across the whole path data takes into the model.
- Track where data came from. Record the source, time and collector of every batch. When something goes wrong, you can then find and drop the bad source.
- Check integrity on download. Store a cryptographic hash of each item when the dataset is built, and reject items whose content no longer matches. The web-scale poisoning paper suggested this as a defense against swapped content on expired domains.
- Lock down data storage. Treat training buckets, labeling tools and feedback tables as production systems. Limit who can write to them, and log every write.
- Limit the weight of any one source. Cap how many examples a single account, domain or labeler can add, so no one source can steer the model.
- Filter and inspect. Remove duplicates and near-duplicates, look for outliers, and run representation checks such as spectral analysis on suspect classes.
- Keep a trusted holdout set. Maintain a small, carefully checked evaluation set that never comes from the open pipeline, and compare every new model against it.
- Probe for triggers. Red teams can search for inputs that flip behavior, try rare tokens and patches, and check narrow slices of data. Red teaming gives a structured way to look for the failure an attacker would plant.
- Keep known-good snapshots. Save versioned copies of data and checkpoints, so you can retrain from a clean point if a source turns out to be poisoned.
Frequently asked questions
Is data poisoning the same as prompt injection?
No. Data poisoning changes what a model learns during training, so the flaw lives in its weights. Prompt injection, and jailbreaks, work by crafting inputs at the moment a trained model runs.
Can a small number of poisoned samples change a model?
Yes. A 2025 pretraining study found that 250 poisoned documents planted a backdoor in language models from 600 million to 13 billion parameters, so the number needed did not grow with the model. Fine-tuning sets are also exposed, because each example carries more weight in a small dataset.
How do model evaluations catch data poisoning?
General benchmarks often miss it, because a poisoned model does well on normal inputs. Evaluations that check narrow slices, search for trigger phrases and compare behavior against a trusted holdout set have a better chance. The basics of how a model evaluation is built apply here: the tasks, the data and the grading all decide what an evaluation can see.
Can data poisoning be reversed after training?
Sometimes, but it is hard to be sure. Further fine-tuning may hide a backdoor without removing it, as the Sleeper Agents results suggest. The surest fix is to find the poisoned data, remove it, and retrain from a clean snapshot.
Why Data Poisoning Matters for AI Safety and Model Evaluations
Data poisoning shows that a model's behavior is only as trustworthy as the data behind it. As models train on more web text, more user feedback and more third-party datasets, the number of people who can influence them grows. Many of those people are never checked.
For AI safety, the hard part is detection. A backdoored model can pass every check that does not include its trigger. That puts weight on evaluations that go beyond average results: slice-by-slice results, deliberate searches for hidden triggers, and checks of internal representations. It also means data provenance and access control are safety work, not only engineering chores.
The practical habit is simple. Treat every route into your training data as an input an attacker could use, and measure the model in ways that would show a planted behavior if one were there.
whitehatstoic
Build it right. Test it thoroughly. Keep it safe. Visit whitehatstoic.com
