Incident Postmortems That Produce Changes Instead of Blame

After an outage, “someone deployed a bad change” can feel like an explanation. It usually describes the final action before failure, not the conditions that allowed the failure to reach customers or made recovery difficult. A useful postmortem reconstructs what happened and improves the system around the people operating it. Blamelessness does not remove accountability. It changes the focus from identifying a person to understanding decisions, safeguards, and constraints. ## Separate evidence from interpretation Build a timeline with timestamps, observed symptoms, actions, and outcomes. Distinguish when the problem began, when the team detected it, and when users recovered. Use logs, deployment records, monitoring, and participant notes. Preserve uncertainty where evidence is missing rather than filling gaps with a confident story after the fact. Describe impact concretely: which workflows failed, which users were affected, and whether data was delayed, lost, or incorrect. A service being technically reachable does not mean customers could complete their tasks. ## Look beyond the triggering event Ask why the change seemed reasonable with the information available at the time. Were test data and production data different? Did an alert point to the wrong component? Did a rollback procedure depend on an unavailable person? Identify contributing factors rather than forcing a single root cause. A database timeout may combine with unlimited retries, insufficient capacity, and an untested recovery path. Record what helped as well. A useful runbook, effective feature switch, or clear support message is worth preserving. Reliability improves by reinforcing successful practices, not only by cataloguing mistakes. ## Make actions specific and verifiable “Be more careful” is not a system improvement. “Add a compatibility test for old clients before removing response fields” addresses a mechanism and can be verified. Assign an owner and define the outcome for each action. Prefer a small number of meaningful changes over a long list nobody can complete. Prioritize reducing impact and recovery time as well as preventing recurrence. Do not let every postmortem default to more alerts. An additional alert is useful only if it detects actionable harm and has an owner who knows what to do. Some incidents need safer defaults, better isolation, or a simpler deployment process instead. Share the report with the teams that need it, removing sensitive customer data and credentials. Revisit whether the actions actually changed behavior or merely closed tickets. The value of a postmortem is not the document itself. It is the next incident being less likely, smaller, easier to diagnose, or faster to resolve because the organization learned something concrete. ## Further reading - [Google SRE: Postmortem culture](https://sre.google/sre-book/postmortem-culture/) - [Google SRE Workbook: Postmortem analysis](https://sre.google/workbook/postmortem-analysis/)