Training Data Memorization Leakage

The model reproduces personal data, secrets, or confidential text it memorized during training or fine-tuning.

Impact

Privacy breaches and secret exposure that cannot be fixed by access controls, because the data lives in the weights.

Why this severity

Impact High × Likelihood Medium = High

Impact high: regulated data disclosure from the weights. Likelihood medium: extraction from a production model is demonstrated in research.

Impact High: Code execution, credential compromise, exposure of regulated or confidential data, irreversible actions, or system-wide failure.
Likelihood Medium: Publicly demonstrated against real products or in peer-reviewed research, or seen in the wild only with a prior foothold such as stolen credentials.

Method: OWASP Risk Rating Methodology

Evidence

Research: Scalable Extraction of Training Data from (Production) Language Models (arXiv 2311.17035, 2023)
Research: Extracting Training Data from Large Language Models (arXiv 2012.07805, 2020)

Exploits

Unscrubbed Fine-Tuning Data Vulnerability · Medium severity

Mitigated by

PII Scrubbing & Privacy-Preserving Training Guardrail · Fixes high risks

Remediation

Scrub PII and secrets from training and fine-tuning data.
Deduplicate data; repeated records are memorized far more often.
Test released models with extraction prompts and canaries.

How to verify

Planted canary strings in training data are not reproducible by the released model.
Membership-inference tests stay near chance on held-out records.

Framework mapping

OWASP Top 10 for LLM Applications 2025

LLM02:2025 Sensitive Information Disclosure

MITRE ATLAS

AML.T0057 LLM Data Leakage
AML.T0024.000 Exfiltration via AI Inference API: Infer Training Data Membership

NIST AI RMF 1.0

MEASURE 2.10 Privacy risk of the AI system is examined and documented