Unscrubbed Fine-Tuning Data

Support tickets, chat logs, or code used for fine-tuning still contain names, credentials, and confidential details.

Impact

Sensitive records become part of the model and can resurface in any user's session.

Why this severity

Impact High × Likelihood Low = Medium

Impact high: personal and secret data enters the weights. Likelihood low: extracting specific records from a fine-tuned model takes many targeted queries.

Impact High: Code execution, credential compromise, exposure of regulated or confidential data, irreversible actions, or system-wide failure.
Likelihood Low: No public cases yet, or the attack needs privileged or insider access.

Method: OWASP Risk Rating Methodology

Evidence

Research: Extracting Training Data from Large Language Models (arXiv 2012.07805, 2020)

Exploited by

Training Data Memorization Leakage Threat · High severity

Mitigated by

PII Scrubbing & Privacy-Preserving Training Guardrail · Fixes high risks

Remediation

Run PII and secret detectors over every training shard.
Prefer synthetic or redacted data for fine-tuning.
Keep a data inventory linking each model to the datasets it saw.

How to verify

Data pipeline blocks shards with detector hits above threshold.
Model card lists datasets and their scrubbing status.

Framework mapping

OWASP Top 10 for LLM Applications 2025

LLM02:2025 Sensitive Information Disclosure
LLM04:2025 Data and Model Poisoning

MITRE ATLAS

AML.T0057 LLM Data Leakage

NIST AI RMF 1.0

MEASURE 2.10 Privacy risk of the AI system is examined and documented
MAP 4.1 Risks of AI components, including third-party data and software, are mapped