Training & Fine-Tune Data Poisoning

Adversary seeds pre-training, fine-tuning, or RLHF data with crafted samples that implant backdoors or skew behavior.

Impact

A trigger phrase silently flips model behavior in production, bypassing safety tuning and evading standard evals.

Why this severity

Impact High × Likelihood Medium = High

Impact high: backdoors survive into production and evade evals. Likelihood medium: practical at web scale per research, and Tay was poisoned in production, but it requires influence over a training source.

Impact High: Code execution, credential compromise, exposure of regulated or confidential data, irreversible actions, or system-wide failure.
Likelihood Medium: Publicly demonstrated against real products or in peer-reviewed research, or seen in the wild only with a prior foothold such as stolen credentials.

Method: OWASP Risk Rating Methodology

Evidence

Incident: Tay Poisoning (MITRE ATLAS case study AML.CS0009, 2016)
Research: Poisoning Web-Scale Training Datasets is Practical (arXiv 2302.10149, 2023)
Research: Poisoning Language Models During Instruction Tuning (arXiv 2305.00944, 2023)
Research: Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (arXiv 2401.05566, 2024)

Exploits

Unverified Data & Model Provenance Vulnerability · High severity

Leads to

Malicious Model Artifacts Threat · Critical severity

Mitigated by

Dataset Provenance & Poison Screening Guardrail · Fixes high risks

Remediation

Track provenance for every dataset shard and fine-tune contribution.
Deduplicate and outlier-filter training data before each run.
Hold out trigger-hunting and behavioral-diff evals between model versions.

How to verify

Train a canary model on a planted-trigger fixture and confirm detectors flag it.
Compare refusal and toxicity rates against the previous release on a frozen suite.

Framework mapping

OWASP Top 10 for LLM Applications 2025

LLM04:2025 Data and Model Poisoning

MITRE ATLAS

AML.T0020 Training Data Poisoning
AML.T0018 Manipulate AI Model

NIST AI RMF 1.0

MEASURE 2.7 AI system security and resilience are evaluated and documented
MANAGE 3.2 Pre-trained models are monitored as part of regular maintenance