Hallucinated Answers Acted On

Users or downstream systems act on fabricated facts, citations, or policies the model states with confidence.

Impact

Wrong decisions with legal, financial, or safety consequences; the operator can be held liable for what its chatbot said.

Why this severity

Impact Medium × Likelihood High = High

Impact medium: usually financial or reputational, higher in medical or legal settings. Likelihood high: a tribunal held Air Canada liable for its chatbot's invented policy, with no attacker involved.

Impact Medium: Limited or recoverable harm: cost, availability, reputational damage, or exposure confined to one session or user.
Likelihood High: At least one documented real-world incident, and the attack needs no access beyond what the system exposes to ordinary users or content (publishing to a public registry counts as ordinary).

Method: OWASP Risk Rating Methodology

Evidence

Incident: Moffatt v. Air Canada, 2024 BCCRT 149: airline held liable for its chatbot's wrong bereavement-fare answer (BC Civil Resolution Tribunal decision (PDF), 2024)

Exploits

Ungrounded, Unverified Answers Vulnerability · High severity
Interface Invites Overreliance Vulnerability · Medium severity

Mitigated by

Grounding & Citation Verification Guardrail · Fixes high risks
Uncertainty Labeling & High-Stakes Review Guardrail · Fixes high risks

Remediation

Ground answers in retrieved, trusted sources and show the citations.
Route high-stakes topics (legal, medical, financial, policy) to human review.
State the system's limits in the interface, not only in the terms of service.

How to verify

Track factual accuracy on a labeled question set for every release.
Spot-check production answers in high-stakes categories each week.

Framework mapping

OWASP Top 10 for LLM Applications 2025

LLM09:2025 Misinformation

MITRE ATLAS

AML.T0048 External Harms

NIST AI RMF 1.0

MEASURE 2.5 AI system is demonstrated to be valid and reliable
MEASURE 2.9 Model is explained and output is interpreted within its context