AI Security Atlas › All entries › LLM Application & Output Security
Jailbreak & Safety Bypass
Role-play, encoding tricks, or many-shot prompting push the model past its safety policy.
Open in the interactive atlas →Impact
The application produces harmful, defamatory, or policy-violating content under its own brand.
Why this severity
Impact Medium × Likelihood High = High
Impact medium: harmful output under the product's brand, usually without system compromise. Likelihood high: Storm-2139 built jailbreak tooling to bypass Azure OpenAI guardrails (2024 incident).
Impact Medium: Limited or recoverable harm: cost, availability, reputational damage, or exposure confined to one session or user.
Likelihood High: At least one documented real-world incident, and the attack needs no access beyond what the system exposes to ordinary users or content (publishing to a public registry counts as ordinary).
Method: OWASP Risk Rating Methodology
Evidence
- Incident: Storm-2139 Azure OpenAI Guardrail Bypass (MITRE ATLAS case study AML.CS0057, 2024)
- Research: Universal and Transferable Adversarial Attacks on Aligned Language Models (arXiv 2307.15043, 2023)
Related
Exploits
- Single-Layer Safety Reliance Vulnerability · Medium severity
Leads to
- Secrets & Policy in System Prompt Vulnerability · Medium severity
Mitigated by
- Input/Output Safety Classifiers Guardrail · Fixes high risks
Remediation
- Layer input and output classifiers independent of the base model.
- Cap conversation length and context stuffing for untrusted users.
- Continuously red-team against fresh jailbreak corpora.
How to verify
- Track attack success rate per release on a frozen jailbreak suite.
- Classifier failures are logged with full transcripts for review.
Framework mapping
OWASP Top 10 for LLM Applications 2025
- LLM01:2025 Prompt Injection
MITRE ATLAS
- AML.T0054 LLM Jailbreak
NIST AI RMF 1.0
- MEASURE 2.6 AI system is evaluated regularly for safety risks