Incident Postmortems

Describe a production incident you handled and how you structured the postmortem. What makes a good blameless postmortem?

Senior questions

SREadvanced

// interview question

Describe a production incident you handled and how you structured the postmortem. What makes a good blameless postmortem?

Sample answer

A blameless postmortem focuses on systems improvement, not individual blame. Structure: incident timeline (what happened), impact assessment (users affected, duration), root cause analysis (5 whys technique), contributing factors, action items with owners and deadlines, and lessons learned. Share widely to prevent similar issues across teams.

Why this matters

Blameless postmortems create psychological safety where engineers report issues honestly. If people fear punishment, they hide problems. Focus questions on 'how did our systems allow this?' not 'who made a mistake?'. This cultural shift is essential for building reliable systems and learning organizations.

Code examples

Postmortem template

markdown

# Incident Postmortem: API Outage 2024-01-15

## Summary
API service was unavailable for 47 minutes due to database connection pool exhaustion.

## Impact
- Duration: 10:15 - 11:02 UTC (47 minutes)
- Users affected: ~12,000 (30% of active users)
- Revenue impact: ~$8,500 in failed transactions

## Timeline
- 10:15 - PagerDuty alert: API error rate > 50%
- 10:18 - On-call engineer begins investigation
- 10:25 - Root cause identified: DB connection pool
- 10:45 - Fix deployed: increased pool size
- 11:02 - All systems normal

## Root Cause
A slow query caused connections to be held longer, exhausting the pool.

## Action Items
1. [ ] Add connection pool monitoring (@alice, due: Jan 20)
2. [ ] Implement query timeout (@bob, due: Jan 22)
3. [ ] Add runbook for connection issues (@charlie, due: Jan 25)

Common mistakes to avoid

Blaming individuals instead of focusing on systemic improvements
Not following up on action items from previous postmortems
Writing postmortems only for major incidents (minor ones teach too)

Likely follow-ups

How do you ensure action items from postmortems actually get completed?
What's the difference between root cause and contributing factors?
How do you balance incident response speed with thoroughness?

Answer out loud first, then check yourself against the model answer.

Practice all Senior questions More SRE questions

#incident-management#postmortem#sre#culture#leadership

Also worth your time on this topic

Interview

Error Budget Management

Your service has a 99.9% availability SLO over a 30-day window. How much downtime does that give you, and what do you actually do with that error budget day-to-day?

mid

Article

How to Build an Effective On-Call Rotation and Escalation Policy

Your phone buzzed at 3:14 AM for a disk warning that auto-resolved by 3:16. Nobody fixes the alert. The next person on rotation hates their life. Here is how to build on-call schedules, escalation policies, and alert rules that respect your engineers.

Checklist

SLOs, SLIs, and Error Budgets: A Practical Implementation Guide

A step-by-step checklist for defining service level objectives, picking the right service level indicators, and using error budgets to make better decisions about reliability vs. feature velocity.

45-90 minutes

Incident Postmortems

More SRE interview questions

Also worth your time on this topic

Error Budget Management

How to Build an Effective On-Call Rotation and Escalation Policy

SLOs, SLIs, and Error Budgets: A Practical Implementation Guide