Incidents, Escalations, and Correction of Errors
Module 3 · Operational Excellence

Incidents, Escalations, and Correction of Errors

The TAM's role during a high-severity event: escalation paths, incident communication, updates for executives, and running a blameless post-incident review using Amazon's Correction of Errors (COE) approach and the 5 Whys.

The TAM During an Incident

When a customer has a business-critical event, the TAM is usually not the one typing commands. The TAM is the connector and communicator:

  • Confirm a case is open at the right severity, with impact, timeline, and resource IDs
  • Escalate inside AWS so the right support and service-team engineers are engaged
  • Join the customer’s incident bridge and keep AWS and the customer in sync
  • Send clear, regular updates to executives on both sides
  • Make sure the event ends in a post-incident review with real actions
  1. 1 Detect

    Alarm, customer call, or AWS Health

  2. 2 Engage

    Case + escalation + bridge

  3. 3 Mitigate

    Restore service first

  4. 4 Communicate

    Regular, timed updates

  5. 5 Resolve

    Confirm recovery with data

  6. 6 Review

    COE; actions with owners


Escalating Well

Escalation is a skill. A good escalation is specific:

“Customer X, Sev 1 case 1234567: checkout errors at 40% since 14:05 UTC in us-east-1, about $50K/hour revenue impact. Errors align with DynamoDB throttling on table orders even though consumed capacity is below provisioned. Need the DynamoDB team to check for a hot partition or a service-side issue. Next customer update at 14:45.”

That message contains who, impact, timeline, evidence, the specific ask, and the next checkpoint. Compare it with “Customer X is really upset, can someone look at their case?”

When to escalate: when the response is too slow for the business impact, when the case needs a service team, when the customer’s executives are involved, or when there’s a pattern across several customers.

How often: escalate early for real business-critical impact. A TAM who protects the customer by escalating early is showing Customer Obsession and Bias for Action. A TAM who escalates everything loses credibility.


Communicating to Executives

During an event, leaders want three things: what’s the impact, what’s being done, and when will I hear next? Use a consistent update format:

FieldExample
StatusMitigating
Impact~40% of checkout requests failing in us-east-1 since 14:05 UTC
Current actionsShifting writes to a new table partition key; DynamoDB team engaged
Next update14:45 UTC, or sooner if status changes

Never guess at a root cause in an executive update. Say “suspected” or “under investigation” until it’s confirmed. And never miss your next update time. If nothing changed, send the update anyway and say so.


Correction of Errors (COE)

Amazon’s post-incident process is called Correction of Errors (COE). TAMs are often asked to help customers run something similar, or to explain an AWS-side event to a customer. A COE typically covers:

  1. What happened: a factual summary
  2. Customer impact: who was affected, how badly, for how long, with data
  3. Timeline: detection, escalation, mitigation, resolution, with times
  4. Metrics: graphs that show the event
  5. 5 Whys: asking “why?” repeatedly to get past the surface cause
  6. Lessons learned
  7. Action items: each with an owner, a priority, and a due date

The 5 Whys, in practice

  • Why did checkout fail? DynamoDB throttled writes.
  • Why? All writes went to one partition.
  • Why? The partition key was the date, so every order today shared a key.
  • Why was that not caught? The load test used random dates.
  • Why? There’s no load-test standard that requires production-like key distribution.
Checkout failed: DynamoDB throttled writesWhy? All writes went to one partitionWhy? The partition key was the dateWhy? The load test used random datesWhy? No standard requires production-like keysSymptomMissing mechanism: fix this

Action items now target the mechanism (the load-test standard), not just the symptom (the key).

⚠️ Common Pitfall: Blame in the Review

Common mistake: the review concludes “an engineer pushed a bad config” and the action item is “be more careful.”

COEs are blameless. People make mistakes; mechanisms should catch them. Good actions add a guardrail: automated config validation, a staged rollout, an alarm, a runbook. Amazon’s phrase is “good intentions don’t work, mechanisms do.”


Key Takeaway

In an incident, a TAM engages the right AWS experts, escalates specifically, and communicates on a schedule. Afterward, they drive a blameless COE: impact, timeline, 5 Whys down to the missing mechanism, and action items with owners and dates, then track those actions to completion.

🧪 Try It: Rewrite a Weak Escalation

Rewrite this message so it would get fast, useful action:

“Hi team, my customer is having problems with their database again and they’re pretty frustrated. Can someone take a look ASAP?”

Include: customer, case number and severity, exact symptom, start time, business impact, evidence so far, the specific ask, and next update time. Make up realistic details if you need to.

🧠 Knowledge Check

1. Which element is most often missing from a weak escalation?

2. There's no new information at your promised update time. What should you do?

3. What is the purpose of the 5 Whys in a COE?

4. Which is the best COE action item?