How to write an incident postmortem for custom software
After an incident on custom software already in production, the postmortem records the timeline, the effect users could see, what stopped that effect, the cause as far as it is known, and who will finish each follow-up by which day. The operations runbook names who answers the page. The rollback plan says how to undo a release. This page is the record kept once the incident has settled.
On this page
- Keep the postmortem apart from the runbook and the rollback plan
- Decide in advance which incidents get a postmortem
- Write the timeline from the record of the time
- Write the effect users could see
- Separate what stopped the effect from the cause known so far
- Give each follow-up an owner and a day
- Write what happened; names are there for the timeline
- The postmortem is finished when someone has reviewed it
Keep the postmortem apart from the runbook and the rollback plan
The operations runbook says, once production is in use, who answers this shift, where the logs and dashboards are, the first check for a known failure, when to call the next person, and when to follow rollback. The rollback plan says which visible symptom can start a reversal, who decides, and whether this reversal restores data or turns a feature flag off. The postmortem answers a different question: after the effect has stopped, or has turned into ongoing work, where the timeline, the effect users could see, the actions taken, the cause as far as it is known, and the owners of the follow-up are written down. The three documents can point at the same incident number. The duty roster is not rewritten as a postmortem, and the rollback triggers are not copied into the cause. See How to write an operations runbook for custom software and How to write a rollback plan for custom software go-live.
The handover still says who holds the repository. A change of specification still goes through change control. See How to prepare a custom software project handover and How to control scope changes in custom software. Delivery still sits under Custom software, websites, and applications. The postmortem cover carries its own version, the incident number, and the path where it is stored. If the runbook already names where postmortems live, this page uses that path. It does not open a second place that only the person on duty knows.
Decide in advance which incidents get a postmortem
Read on 27 October 2026, Google SRE: Postmortem Culture is written by John Lunney and Sue Lueder and edited by Gary O'Connor. The page shows no publication date. It defines a postmortem as a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause or causes, and the follow-up actions to prevent the incident from recurring. The page says the goals are to document the incident, to understand the contributing causes, and to put preventive actions in place so that a recurrence is less likely or less severe. It also says the criteria should be defined before an incident, so that everyone knows when a postmortem is required, and that any stakeholder may request one for an event.
The common triggers on that page are the page's account of Google: user-visible downtime or degradation beyond a threshold, data loss of any kind, intervention by the person on call (including a release rollback or a reroute of traffic), a resolution time above a threshold, and a monitoring failure that meant the incident was found by hand. The page does not print a number that a custom project must adopt. The company's own criteria go on the cover of the postmortem procedure, with who agreed them and on which day: which user-visible symptom requires a write-up, whether data loss requires one, and whether a rollback or a reroute by the person on duty requires one. If the cover has no date, a stakeholder can still ask for a write-up of this incident, and the procedure itself gets a date before anyone copies an example from the page onto the cover.
Write the timeline from the record of the time
Read on 27 October 2026, Google SRE: example postmortem shows no publication date. The example on the page is dated 2015-10-21 and uses a teaching search service as its subject. The page calls the timeline a screenplay of the incident and says to start from the timeline in the incident record, then add other relevant entries. Times name the time zone. Each line is something that happened: when the page arrived, who declared the incident, who did which step, when the effect stopped, and when the incident ended. The line cites the log, the dashboard, the paging record, or the message kept at the time.
The minutes, query counts, and revenue wording in the example stay on that page. They are not copied into this custom incident. A moment the logs do not show is written "not in the logs". A round number is not supplied in its place. If a spoken recollection and the log disagree, both lines stay, with who compared them and on which day. The timeline is not rewritten as the cause. The cause is the next section. The names of the people on duty stay on the timeline, because the timeline is who did which step.
Write the effect users could see
The culture page includes impact among the things a postmortem records. A note on the example page says impact is the effect on users, revenue, and the like. The custom postmortem records what users could see: which function was unavailable or slow, from which moment on the timeline to which moment, and which roles already using the system were affected. The moments are the ones that can be read from a log or a dashboard. If the duration cannot be read, the field says "duration not in the record". A guessed number is not entered.
If revenue, compensation, or a penalty was not checked for this incident, the field says "not recorded". The example page's sentence about revenue is not rewritten as zero or as an amount. Information that could identify an end user does not go into the postmortem. The culture page says that kind of information stays out of internal documents as well. The affected scope is written as a role, a function, or a count, and the count points at the screen it was read from, with the day it was read.
Separate what stopped the effect from the cause known so far
The culture page treats the actions taken to mitigate or resolve the incident as separate from the cause. The postmortem therefore has two fields. The first says what stopped the effect at the time: who did which step, and when, such as returning to the previous release, turning a switch off, moving traffic to an environment that still worked, or stopping writes. If the step followed the rollback plan, the field points at that version and at the person who decided. It does not restate the triggers. See How to write a rollback plan for custom software go-live. The second field says the cause as far as it is known: which conditions came together, and how far the inquiry has gone.
A note on the example page says the cause explains the circumstances in which the incident happened, and that a technique such as the Five Whys is often helpful for the contributing factors. That is the page's suggestion. It does not require a custom project to use that same series of questions. Each step that is known is its own line, with the log or the change record it rests on. What is not yet known is written "not yet known", with who will continue and by which day. A sentence that sounds complete and has no source is not added. The culture page lists whether the cause was deep enough as a review question. If it was not, the review asks for more inquiry. It does not close the incident by shortening the sentence.
Give each follow-up an owner and a day
The culture page says preventive actions belong in the work, so that a recurrence is less likely or less severe. The example's action table has the action, a type, an owner, and a tracking id. The types in the example include mitigate, prevent, process, and other. A note on that page says a first-impulse action is often too large or too costly and may need to be narrowed, and that there is a risk of over-fitting one incident, for example by adding an alert when an earlier test could have caught the problem. The custom postmortem keeps one owner on every action. The example table does not print a due date as a fixed column. This incident writes one: who will finish by which day, and where the result will sit.
An action with no owner, or with no day, stays unset. It is not yet scheduled work. If the owner is away, the deputy is already named on the line. An action that would change the specification opens a change request, and the postmortem only points at that number. See How to control scope changes in custom software. An action that changes the first check in the runbook points at that entry and the date of the edit. The runbook is not copied into the postmortem. The tracking id is the work item or defect the company already uses. If there is no id, the line names the path where the action is kept.
Write what happened; names are there for the timeline
The culture page says a postmortem looks for contributing causes and does not indict a person or a team for bad behaviour. Written that way, it assumes the people involved meant well and did the reasonable thing with the information they had. The page says that if the write-up becomes a search for who was wrong, people will keep issues quiet because they expect punishment. The postmortem still says who did which step, and when, because the timeline needs those names. The names match the record. They are not a disciplinary note. Discipline, appraisal, and bonuses stay off this document.
The page also says that prevention becomes writable when the question turns from blame to why the information at the time was incomplete or wrong. A person is not rewritten into a different person. The system and the procedure can be changed so that the next design or maintenance choice is easier to get right. A remark about a person is put back, in review, onto the event and the information: which dashboard, which runbook entry, and which alert were visible. After the remark is removed, the event and the follow-up remain.
The postmortem is finished when someone has reviewed it
The culture page says an unreviewed postmortem might as well never have existed. It lists questions a review can ask: whether the key incident data was kept, whether the impact is complete, whether the cause is deep enough, whether the action plan is suitable and the related fixes have a sensible priority, and whether the outcome was shared with the people who need it. The custom postmortem uses those questions as the review list. The reviewer signs and dates it. The reviewer is not assumed to be the only author. The author is named as well. One person may fill both lines. The lines stay separate.
If the timeline, the effect users could see, what stopped the effect, the cause or "not yet known", or an owner and a day on each follow-up is blank, the status stays draft. A draft can still be edited. After the reviewer signs, the postmortem is placed at the path on the cover, and that path is written back on the incident number. The share list says who received it and which existing channel carried it. End-user personal data does not go out with the postmortem. The status becomes reviewed when three things are true: the data and the impact are written, the cause is written as far as it is known and someone is still looking at the open part, and every follow-up has an owner and a day.