AI Security Atlas › All entries › Autonomous Agent & MCP Security
Goal Alignment Drift
Long-running agents gradually reinterpret objectives after poisoned memory, competing rewards, or multi-turn adversarial framing.
Open in the interactive atlas →Impact
Agents pursue shadow goals that violate policy while still appearing task-complete in telemetry.
Why this severity
Impact High × Likelihood Medium = High
Impact high: the agent acts against policy while appearing healthy. Likelihood medium: observed in a 2026 incident where evaluation agents crossed their boundaries, but it needs long-running autonomy.
Impact High: Code execution, credential compromise, exposure of regulated or confidential data, irreversible actions, or system-wide failure.
Likelihood Medium: Publicly demonstrated against real products or in peer-reviewed research, or seen in the wild only with a prior foothold such as stolen credentials.
Method: OWASP Risk Rating Methodology
Evidence
- Incident: Autonomous OpenAI Evaluation Agents Compromise Hugging Face Infrastructure (MITRE ATLAS case study AML.CS0068, 2026)
- Demonstration: Hacking ChatGPT's Memories with Prompt Injection (MITRE ATLAS case study AML.CS0040, 2024)
Related
Exploits
- Dynamic Code Execution Vulnerability · High severity
Leads to
- Unvalidated Function Arguments Vulnerability · High severity
- Rogue or Compromised Agent Threat · High severity
Mitigated by
- Dual-Agent Approval Loop Guardrail · Fixes high risks
- Agent Behavioral Monitoring & Kill Switch Guardrail · Fixes high risks
Remediation
- Checkpoint goals against a signed policy charter each turn.
- Separate planner and executor agents with dual-approval on goal changes.
- Expire and re-hash agent memory entries with integrity checks.
How to verify
- Inject contradictory goal updates mid-run and confirm charter veto.
- Review divergence metrics between declared goal and tool sequence.
Framework mapping
OWASP Top 10 for LLM Applications 2025
- LLM01:2025 Prompt Injection
- LLM06:2025 Excessive Agency
OWASP Top 10 for Agentic Applications 2026
- ASI01:2026 Agent Goal Hijack
- ASI10:2026 Rogue Agents
MITRE ATLAS
- AML.T0051 LLM Prompt Injection
NIST AI RMF 1.0
- MEASURE 2.4 AI system functionality and behavior are monitored in production