Incident Management and Postmortems
Seniors are expected to lead incidents and turn them into lasting improvements. Learn the incident lifecycle and severities, the roles in an incident response, how to run a blameless postmortem, and how to tell an incident story in an interview.
🎯 What you'll learn
- The incident lifecycle and severity levels
- Incident roles, especially the incident commander
- Blameless postmortems and action items
- Telling an incident story
📚 Before you start
Seniors are expected to lead incidents, not just fix bugs during them, and to turn incidents into lasting improvements through blameless postmortems. Interviewers ask "Tell me about an incident you handled" in almost every senior loop.
1The Incident Lifecycle
- Detect: an alert (ideally on symptoms such as error rate or latency SLOs, not causes), a customer report, or a failed deploy.
- Triage: declare an incident, assign a severity, and open a channel.
- Mitigate: stop the bleeding first: roll back, fail over, disable a feature flag, scale up, shed load. Root cause can wait.
- Resolve: confirm metrics are back to normal; keep watching.
- Postmortem: within a few days, blameless, with a timeline and action items.
- Follow up: action items get owners and dates, and are actually done.
| Severity | Example | Response |
|---|---|---|
| SEV1 | Checkout down for all users; data loss | All hands, incident commander, exec updates every 30 min |
| SEV2 | Major feature degraded for some users | On-call plus team, updates every hour |
| SEV3 | Minor degradation, workaround exists | Normal working hours |
2Roles
| Role | Responsibility |
|---|---|
| Incident commander (IC) | Coordinates; decides; doesn't debug. Keeps the team focused on mitigation. |
| Operations / subject experts | Investigate and apply fixes. |
| Communications lead | Status page, stakeholder updates on a fixed cadence. |
| Scribe | Keeps a timeline of actions and findings in the channel. |
In small teams one person may hold several roles, but the IC role should be explicit.
3Blameless Postmortems
A postmortem asks how the system allowed the failure, not who to blame. People act reasonably given the information they had; if a single person's mistake could take production down, the system lacked a guardrail.
Contents:
- Summary: what happened, impact (users, duration, revenue), severity.
- Timeline: detection, key decisions, mitigation, resolution, with timestamps.
- Root cause and contributing factors: use "5 whys" or a causal diagram; there's usually more than one cause.
- What went well / what went poorly, including detection and response.
- Action items: specific, owned, dated, prioritized (prevent, detect faster, mitigate faster).
Track MTTD (time to detect), MTTR (time to restore) and repeat incidents. Many teams also review postmortems across the organization to spot patterns.
4Telling an Incident Story
Use STAR(L), with emphasis on your role (IC? the person who found the cause?), mitigation first, communication, and the systemic fix. Numbers: duration, users affected, MTTR before and after your improvements.
⚠️ Common Pitfalls
- Debugging before mitigating: the root-cause hunt extends the outage.
- No IC: five people debugging in parallel, nobody coordinating or communicating.
- Blame: "an engineer ran the wrong command" ends the analysis where it should start.
- Action items that never get done, or vague ones ("be more careful").
- Silence to stakeholders: no updates for an hour during a SEV1.
📋 Quick Reference
Detect → triage (severity, channel) → MITIGATE (rollback/flag/failover) → resolve → postmortem → follow up
Roles: incident commander (coordinates, doesn't debug) · experts · comms · scribe
Blameless: how did the system allow it? Several contributing causes, specific owned action items
Metrics: MTTD, MTTR, repeat incidents🧠 Key takeaways
- Mitigate first (roll back, flag off, fail over); find the root cause later.
- Make the incident commander explicit: they coordinate and communicate, not debug.
- Run blameless postmortems: ask how the system allowed the failure.
- Action items must be specific, owned and dated; track MTTD and MTTR.
🛠️ Practice exercises
Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.
✅ Check yourself
Answer the questions — feedback is instant.