How is data poisoning different from prompt injection?

Prompt injection happens at inference—the attacker manipulates the input the model sees in a single request. Data poisoning happens before inference—the attacker corrupts the training data, fine-tuning corpus, RAG knowledge base, or tool outputs the model relies on. Prompt injection failures are visible in the current request; poisoning failures are baked into the model or its data and surface as systematic, hard-to-debug misbehavior. Both are on the OWASP Top 10 for LLM Applications (LLM01 and LLM04 respectively) and most production agents need defenses for both.

How much poisoning does it take to compromise an LLM?

Less than most teams assume. Anthropic's October 2025 study found that 250 malicious documents reliably backdoor LLMs across model sizes from 600M to 13B parameters—0.00016% of training tokens at the largest scale. The result overturns the prior assumption that poisoning needs to scale with model size; the absolute count of poisoned documents is what matters, not the percentage.

What's the best defense?

Layered. Verify model and data provenance (signed datasets, SafeTensors over pickle, hash allowlists). Run training-data anomaly detection (Activation Clustering, Isolation Forest, robust aggregation in federated setups). Add inference-time activation analysis for RAG (RevPRAG-style detection, ~98% TPR / ~1% FPR per EMNLP 2025). Cap tool privileges and require approval for irreversible actions. Plant canary phrases in your training data so you can detect a successful poisoning attempt. No single defense is sufficient—frontier labs themselves acknowledge poisoning and indirect injection cannot be fully eliminated at the model layer in current architectures.

Data Poisoning

Written by Max Zeshut

Founder at Agentmelt · Last updated Sep 9, 2026

An integrity attack on AI systems in which an adversary tampers with the data a model trains, fine-tunes, retrieves, or otherwise consumes—so the resulting behavior is bent in the attacker's favor (a backdoor, a bias, a wrong recommendation, or gibberish output triggered by a specific phrase). Unlike Prompt Injection (which targets the model at inference), data poisoning corrupts the substrate before deployment, and the corruption usually survives standard evaluation because the model behaves normally on everything except the attacker's trigger conditions. Modern poisoning hits five surfaces: pre-training, fine-tuning, RAG corpus, model supply chain, and tools/agents (see Tool Poisoning). OWASP tracks it as LLM04:2025. Anthropic's 2025 study showed that as few as 250 malicious documents reliably backdoor LLMs from 600M to 13B parameters, regardless of model size.

Example

A retail company runs a RAG-based AI support agent over its public help center. An attacker submits 30 carefully crafted articles to the help center's user-suggested-content channel that contain instructions invisible in normal reading but parsed by the embedding model. After the next index refresh, customers asking about refunds get a response that quietly recommends a competitor product. The team only catches it weeks later when CSAT drops; a RAG traceback framework eventually locates the poisoned chunks. The fix is dual: tighten the user-content review gate, and add an activation-based detector on every retrieval to flag anomalous influences in real time.

Frequently asked questions

How is data poisoning different from prompt injection?: Prompt injection happens at inference—the attacker manipulates the input the model sees in a single request. Data poisoning happens before inference—the attacker corrupts the training data, fine-tuning corpus, RAG knowledge base, or tool outputs the model relies on. Prompt injection failures are visible in the current request; poisoning failures are baked into the model or its data and surface as systematic, hard-to-debug misbehavior. Both are on the OWASP Top 10 for LLM Applications (LLM01 and LLM04 respectively) and most production agents need defenses for both.
How much poisoning does it take to compromise an LLM?: Less than most teams assume. Anthropic's October 2025 study found that 250 malicious documents reliably backdoor LLMs across model sizes from 600M to 13B parameters—0.00016% of training tokens at the largest scale. The result overturns the prior assumption that poisoning needs to scale with model size; the absolute count of poisoned documents is what matters, not the percentage.
What's the best defense?: Layered. Verify model and data provenance (signed datasets, SafeTensors over pickle, hash allowlists). Run training-data anomaly detection (Activation Clustering, Isolation Forest, robust aggregation in federated setups). Add inference-time activation analysis for RAG (RevPRAG-style detection, ~98% TPR / ~1% FPR per EMNLP 2025). Cap tool privileges and require approval for irreversible actions. Plant canary phrases in your training data so you can detect a successful poisoning attempt. No single defense is sufficient—frontier labs themselves acknowledge poisoning and indirect injection cannot be fully eliminated at the model layer in current architectures.

See it as a workflow

Security Alert Triage Workflow

Trigger, steps, n8n nodes, guardrails and an importable template — plus what it costs to have it built.

Or skip the build

Workflows from $197/month, custom agents from $2,000.

Related glossary terms

Related niches

Back to glossary

Loading…